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Summary 


Background 

All  applicants  for  military  enlistment  are  selected  and  classified  by  using  the  scores  on 
the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB).  Previous  studies  by  the 
Army,  Air  Force,  and  Navy  suggested  that  the  ASVAB  might  have  increased  validity  if 
new  tests  were  added  or  substituted  for  the  existing  ones,  particularly  computerized  tests 
of  abilities  not  easily  measured  by  paper-and-pencil  tests.  The  utility  of  a  3%  increase  in 
validity  had  been  estimated  to  be  in  the  tens  of  millions  of  dollars  from  the  resulting 
improvement  in  selection  and  classification.  Accordingly,  the  military  services,  under  the 
guidance  of  DOD,  undertook  a  large-scale  validation  of  the  most  promising  experimental 
tests  developed  by  the  services’  personnel  laboratories.  This  joint-service  battery  of  nine 
experimental  tests  was  called  the  Enhanced  Computer  Administered  Testing  (ECAT) 
battery. 

Objective 

The  ECAT  project  was  designed  to  estimate  how  much  validity  gain  could  be  obtained 
from  adding  the  ECAT  tests  to  the  ASVAB,  determine  which  tests  were  most  promising, 
and  determine  in  what  military  occupational  specialities  the  gains  were  the  largest 

Approach 

Over  10,000  recruits  scheduled  for  training  in  3  Army  schools,  2  Air  Force  schools,  or 
13  Navy  schools  were  administered  the  ECAT  battery.  Seventy-seven  criteria  for  training 
performance  were  collected  among  the  18  schools,  including  many  hands-on  performance 
measures.  The  validity  of  the  ASVAB  tests  in  a  multiple  regression  equation  was 
compared  with  the  validity  of  the  combined  ECAT  and  ASVAB  battery.  Four  kinds  of 
analyses  were  done  for  incremental  validity  in  terms  of  (1)  general  ability,  (2)  ability 
factors,  (3)  individual  tests,  and  (4)  unit-weighted  selector  composites. 

Results 

Working  memory  and  spatial  ability  tests  produced  large  increases  in  validity  for 
predicting  Air  Traffic  Control  training  performance  in  two  Air  Force  samples  and  one 
Navy  Sample.  Psychomotor  tracking  tests  and  spatial  ability  tests  greatly  increased 
prediction  of  Heavy  Antiarmor  Weapons  firing  accuracy. 

Using  all  ECAT  tests,  six  of  13  Navy  schools  showed  significant  increase  in  validity. 
Averaged  over  all  schools,  the  prediction  of  hands-on  performance  increased  over  5%, 
while  the  prediction  of  School  Grades  improved  only  2%. 

About  75%  of  the  incremental  validity  of  ECAT  can  be  attained  by  using  just  three  of 
the  nine  tests:  Two-Hand  Tracking,  Mental  Counters,  and  Assembling  Objects,  each  of 
which  represents  a  different  ability  factor. 
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Conclusions 


Many  ECAT  tests  have  substantial  simple  validities  for  predicting  school  performance. 
In  some  military  training  courses,  the  ASVAB’s  prediction  of  school  practical 
performance  can  be  substantially  improved  by  using  ECAT  tests  in  optimally-weighted 
composites.  Validity  increases  are  greatest  (averaging  5.7%)  when  laboratory  or  simulator 
performance  criteria  are  used,  rather  than  school  grades  (averaging  1.7%).  Increases  for 
some  schools  are  much  larger  than  this,  while  other  schools  have  no  significant  validity 
improvement.  Factor  scores  are  more  than  98%  as  valid  as  individual  tests  in  multiple 
regression,  but  relying  on  “g”  alone  reduces  validity  by  as  much  as  8.9%  on  the  average. 
ECAT  tests  can  be  used  to  broaden  the  estimate  of  general  mental  ability,  or  “g”  produced 
by  the  ASVAB.  This  enhanced  “g”  has  validity  increments  for  predicting  practical 
performance  criteria  which  are  nearly  as  large  as  the  validity  increments  from  using  all 
tests  in  multiple  regression.  Existing  selector  composites  can  be  improved  by  adding 
ASVAB  tests  to  them.  In  many  cases,  the  validity  improvements  from  doing  so  exceed 
those  from  adding  an  ECAT  test  with  unit  weights. 

Recommendations 

1.  Consideration  should  be  given  toward  the  eventual  incorporation  into  ASVAB  of  a 
Spatial  Ability  measure,  such  as  Assembling  Objects. 

2.  If  CAT-ASVAB  is  universally  implemented,  then  consideration  should  be  given 
toward  including  computerized  tests  of  working  memory,  such  as  Mental  Counters. 

3.  The  Mental  Counters  test  should  be  considered  for  supplementary  administration  to 
potential  students  in  the  Air  Force  and  Navy  Air  Traffic  Control  schools. 

4.  The  Two-Hand  Tracking  test  should  be  considered  for  supplementary  administration 
to  potential  students  in  the  Army  Heavy  Antiarmor  Weapons  school  (11H).  Its 
cost/benefits  for  wider  operational  testing  should  be  evaluated  under  different 
concepts  of  operations. 

5.  A  variety  of  alternative  tracking  tests  should  be  investigated,  to  determine  if  a  mouse, 
trackball,  or  other  off-the-shelf  equipment  could  serve  as  well  as  slide  potentiometers 
and  joysticks.  Human  factors  work  on  alternative  tracking  item  types  and  screen 
displays  should  be  supported. 

6.  Development  of  alternate  forms  and/or  adaptive  item  pools  should  be  started  for  the 
most  promising  ECAT  tests. 

7.  The  most  promising  ECAT  tests  should  be  normed. 

8.  Research  on  optimal  non-negative  weighting  of  ASVAB  tests  for  maximal  cross- 
validated  classification  efficiency  should  be  given  high  priority.  Operational  selector 
composites  eventually  should  be  replaced  by  these  optimal  weighting  methods. 

9.  Military  training  schools  should  be  encouraged  to  incorporate  continuously-scored 
practical  performance  measures  in  their  intermediate  and  final  grades.  The  statistical 
properties  of  Final  School  Grades,  including  reliability  and  validity,  should  be 
continuously  monitored  and  updated,  particularly  following  any  shift  in  curricula. 
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Introduction 


Purpose 

This  report  describes  the  Enhanced  Computer  Administered  Test  (ECAT)  battery  and  the 
results  from  the  Joint-Service  validity  study  that  evaluated  the  battery,  which  was  the  largest 
validation  of  a  single  computerized  test  battery  ever  undertaken.1  ECAT  has  a  significance  beyond 
this  simple  fact;  it  may  fundamentally  alter  the  future  of  military  aptitude  testing.  To  better 
understand  the  importance  of  ECAT,  including  how  it  came  about,  the  first  section  of  this  report 
places  ECAT  in  the  larger  context  of  the  last  decade  of  military  aptitude  research.  The  second 
section  describes  subjects  in  the  validity  study,  the  content  of  the  ECAT  battery  and  its  factor 
structure.  ECAT  is  then  contrasted  with  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB),  which  is  the  present  military  selection  battery.  The  results  of  the  ECAT  validity  study 
are  then  described  in  some  detail,  followed  by  conclusions. 


Historical  Background 

We  are  nearing  the  culmination  of  a  10-year  wave  of  research  on  military  manpower  testing, 
the  largest  since  World  War  II.  The  impetus  for  this  research  began  in  1973,  following  the  end  of 
the  Vietnam  War  when  the  draft  was  terminated  and  the  Services  reverted  to  an  All-Volunteer 
Force.  Over  the  next  decade  the  quality  of  service  applicants  declined  severely  (Eitelberg, 
Laurence,  Waters,  &  Perelman,  1984),  as  did  the  quantity  (Ramsberger  &  Means,  1987;  Laurence 
&  Ramsberger,  1991).  Further  complicating  the  grim  manpower  outlook  was  the  military's 
tremendous  technological  modernization,  a  development  placing  even  greater  intellectual  demands 
on  the  average  enlistee.  Yet  even  as  the  military's  need  for  talented  people  grew  more  acute,  our 
tools  for  identifying  talent  (i.e.,  aptitude  tests)  came  under  growing  attack  from  critics.  In  1978, 
the  Uniform  Guidelines  on  Employee  Selection  Procedures  (EEOC,  1978)  were  adopted  by  the 
Federal  Government —  an  action  that,  in  part,  led  to  the  1981  Congressional  directive  requiring 
that  the  Services  better  document  the  relationship  between  education,  test  scores,  and  actual  job 
performance. 

Collectively,  these  forces  fostered  a  modem,  resurgent  interest  in  military  manpower  testing 
beginning  in  1981.  The  focus  of  these  efforts  was  to  improve  the  ASVAB  since  it  was  the  sole 
military  selection  battery.  There  were  two  main  ASVAB  improvement  themes.  One  was  an  effort 
to  reformat  the  ASVAB  and  improve  the  measurement  properties  of  the  battery.  The  other  theme 
was  to  add  new  aptitude  constructs  and  thus  broaden  the  ASVAB.  The  underlying  logic  was 
straightforward.  If  we  had  more  accurate  and  greater  information  about  an  applicant's  intellectual 


While  this  was  a  Joint-Service  study  that  received  technical  guidance  from  several  Joint-Service  committees,  as  the  lead  Service  laboratory  the  work 
was  conducted  by  the  Navy  Personnel  Research  and  Development  Center.  In  particular,  Kathleen  E.  Moreno  was  solely  responsible  for  the  data 
collection  effort;  she  let  and  monitored  the  data  collection  contract,  obtained  all  the  necessary  military  command  approvals  for  testing,  insured  that 
equipment  was  manufactured  and  available,  that  test  administrators  were  hired  and  trained,  and  throughout  the  conduct  of  the  study  she  maintained  an 
intimate  relationship  with  the  telephone,  handling  daily  crises  and  logistics.  Without  her  dedicated  hard  work,  there  would  have  been  no  ECAT  study. 
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strengths  and  weaknesses,  then  the  military  would  be  better  able  to  select  and  place  applicants; 
psychometrically,  this  amounts  to  increasing  validity.  By  increasing  validity,  it  was  expected  that 
job  performance  would  increase,  job  satisfaction  would  increase,  attrition  would  be  reduced,  and 
collectively  the  military  would  be  more  capable. 


Computerized  Adaptive  Testing  Version  of  the  Armed  Services  Vocational  Aptitude 

Battery:  CAT-ASVAB 

The  ASVAB  reformatting  project  was  CAT-ASVAB  which  formally  began  in  1979  when  the 
Navy  Personnel  Research  and  Development  Center  was  designated  as  Lead  Laboratory  for 
Computerized  Testing.  The  program's  approach  was  shaped  by  two  technological  advances:  the 
availability  of  powerful  microcomputers  and  developments  in  statistical  theories  of  test  scores. 
Blending  these  two  advances  produced  the  concept  for  CAT-ASVAB  which  was  to  develop  a 
computer  administered  version  of  the  ASVAB  redesigned  using  modem  psychometric  theory, 
referred  to  as  Item  Response  Theory.  By  combining  Item  Response  Theory  with  computer 
administration,  the  ASVAB's  power  tests  could  be  made  adaptive  by  specifically  selecting  test 
items  for  an  examinee  based  on  his  or  her  previous  responses.  Adaptive  test  administration  could 
reduce  test  length  by  as  much  as  one-half  while  improving  reliability,  particularly  in  the  extremes 
of  score  distributions  where  applicant  discrimination  was  poor.  Administratively,  computer-based 
testing  could  improve  test  security,  reduce  scoring  errors,  and  provide  immediate  feedback  to 
examinees  and  their  recruiters. 

Although  adaptive  tests  had  been  a  theoretical  possibility  for  a  number  of  years,  no  one  had 
ever  successfully  produced  an  adaptive,  multiple  aptitude  test  battery  intended  for  large-scale  use. 
As  such,  hundreds  of  difficult,  pragmatic,  and  unanticipated  problems  had  to  be  solved  in  the 
development  of  a  working  CAT-ASVAB  system.  There  were  problems  such  as  how  the  system 
should  be  organized,  what  fail-safe  and  failure-recovery  procedures  should  be  included,  what 
hardware  and  networking  system  should  be  chosen,  how  items  should  be  protected,  and  how  the 
frequency  of  item  use  should  be  controlled.  These  and  many  more  problems  were  solved  to 
produce  a  functional  delivery  system. 

More  importantly  though,  there  were  several  critical  research  questions  that  had  to  be 
answered  as  a  prelude  to  operational  use.  One  issue  was  a  concern  that  the  medium  of 
administration  alone  (i.e.,  paper-and-pencil  (P&P)  versus  computer),  would  produce  important 
differences  in  test  items  and  scores.  A  large  scale  1987  study  explicitly  addressed  this  concern 
and  found  it  to  be  generally  unwarranted  (Hetter,  Segall,  &  Bloxom,  1992).  A  second  concern 
was  that  test  score  intercorrelations,  within  and  across  mediums  of  administration,  would  differ 
markedly.  This  issue  was  addressed  in  an  important  1988  study  (Moreno  &  Segall,  1992)  and  the 
results  clearly  demonstrated  that  there  were  no  substantial  differences  among  the  intercorrelation 
matrices  of  ASVAB  tests,  either  within  or  across  test  mediums. 

Having  solved  the  practical  testing  issues  and  assuaged  the  concerns  of  many 
psychometricians  and  policy  makers,  a  final  step  was  required  before  CAT-ASVAB  could  actually 
be  used.  Specifically,  conversion  or  equating  tables  were  required  that  would  allow  CAT- 
ASVAB  and  P&P  ASVAB  scores  to  be  used  interchangeably.  In  1988,  an  elaborately  designed 
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study  was  conducted,  requiring  data  collection  in  several  sites  across  the  country.  The  data  were 
used  to  develop  preliminary  tables  equating  CAT  and  ASVAB  test  scores.  However,  since  the 
original  equating  tables  were  based  on  individuals  who  were  required  to  take  several 
nonoperational  versions  of  the  ASVAB,  the  validity  of  the  equating  tables  had  to  be  verified  in 
one  final  study.  This  study  was  initiated,  and  as  a  result,  in  September  1990,  CAT-ASVAB  was 
operationally  used  for  the  first  time.  CAT-ASVAB  has  become  the  first  operational,  computer 
administered,  adaptive  selection  and  classification  battery  in  use. 


Enhanced  Computer  Administered  Tests:  ECAT 

While  efforts  to  reformat  the  ASVAB  were  focused  and  localized,  attempts  to  broaden  the 
abilities  measured  by  the  ASVAB  were  dispersed,  with  each  of  the  services  conducting  research. 
In  1981  the  Army's  Project  A  was  commissioned  with  a  very  broad  charter  and  sweeping 
objectives.  In  the  same  time  frame,  the  Air  Force's  Learning  Abilities  Measurement  Project 
(LAMP)  began  with  the  goal  of  developing  new  predictors  of  learning.  Smaller  testing  programs 
were  also  started  in  the  Navy. 

Several  common  contextual  stimulants  independently  shaped  the  Services'  attempts  to 
broaden  the  ASVAB,  producing  similarities  in  their  research  programs.  For  example,  the 
availability  of  inexpensive  microcomputers  and  the  momentum  behind  the  computerization  of  the 
ASVAB,  led  the  Services  to  develop  new  tests  that  were  primarily  computer-based.  Moreover, 
the  cognitive  Zeitgeist  in  American  psychology  during  the  mid-1970s  and  1980s  strongly 
influenced  the  programs.  For  example,  all  of  the  Services  investigated  the  use  of  reaction  time 
measures;  the  Air  Forces'  program  was  built  around  a  cognitive  model;  and,  the  Navy's  research 
was  driven  by  cognitive  theories  of  aptitude,  working  memory,  and  mental  imagery. 

Though  there  were  commonalties  across  Services,  there  was  little  collaboration.  However, 
as  work  on  CAT-ASVAB  progressed  and  a  national  renorming  of  ASVAB  was  anticipated, 
additional  impetus  was  provided  to  the  possibility  of  adding  new  aptitude  dimensions  to  the 
ASVAB.  In  December  of  1988,  the  Office  of  the  Assistant  Secretary  of  Defense  (Force 
Management  and  Personnel)  (OASD/FM&P)  redirected  the  CAT-ASVAB  program  to  "include  a 
Joint-Service  validation  of  the  Services'  new  computerized  cognitive  and  psychomotor  tests" 
(Sellman,  1988).  This  directive  was  in  recognition  of  two  facts.  First,  an  early  cost-benefit  study 
suggested  that  fielding  a  computer  version  of  the  ASVAB  may  not  be  cost  effective  relative  to  the 
P&P  version.  (This  assessment  has  since  been  rendered  obsolete  by  the  plummeting  prices  of 
computer  technology  and  by  subsequent  experience  with  an  operational  CAT  system.).  Second, 
other  research  indicated  that  broadening  the  ASVAB's  ability  measures  could  result  in  large 
improvements  in  productivity  per  accession  (Schmidt,  Hunter,  &  Dunn,  1987).  Combining  these 
findings,  a  new  computer-based  ASVAB  augmented  with  new  ability  measures  could  produce  a 
better  and  cost  effective  selection  and  classification  system.  Just  as  importantly  though,  the 
directive  was  a  realization  that  if  decisions  were  to  be  made  about  the  usefulness  of  new  ability 
measures,  they  needed  to  be  evaluated  in  a  single  study  using  the  most  probable  delivery  system 
for  a  computerized  ASVAB,  the  CAT-ASVAB  system.  This  formally  integrated  the  two  research 
strains  to  improve  the  ASVAB. 
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In  response  to  OASD's  redirection,  the  Technical  Advisory  Selection  Panel  (TASP)  was 
established  in  January  of  1989  to  evaluate  and  select  tests  for  the  Joint-Service  validation  battery. 
The  panel's  charter  was  to  select  the  best  tests  in  terms  of  their  psychometric  properties  and 
theoretical  justifications  within  the  constraint  that  the  battery  could  not  exceed  three  hours. 
Across  Services,  hundreds  of  pages  of  documentation  were  submitted  supporting  the  use  of 
dozens  of  new  aptitude  measures2.  Nine  tests  were  chosen  and  combined  into  a  batteiy  named 
ECAT.  A  research  design  was  approved,  the  necessary  software  and  hardware  were  developed 
and/or  acquired,  and  in  February  1990  the  study  began.  Twenty-one  months  later,  testing  ended. 
The  sample  included  enlisted  personnel  in  the  Army,  Navy,  and  Air  Force  representing  18  Military 
Occupational  Specialties  (MOS).  (Additional  details  will  be  provided  in  later  sections.) 


Enhanced  Computer  Administered  Test  Validity  Study 


Enhanced  Computer  Administered  Test  Sample 

Over  11,700  enlisted  personnel  were  tested  with  the  ECAT  battery  in  the  Navy,  Army,  and 
Air  Force.  Individuals  were  tested  prior  to  entering  training  in  one  of  18  different  MOSs  ( these 
will  be  described  later).  The  sample  was  95.5%  male  and  97.5%  used  English  as  their  dominate 
language.  Nearly  84%  of  the  sample  had  obtained  a  high  school  diploma,  an  additional  6.7%  had 
at  least  some  college  level  schooling;  only  9.5%  failed  to  complete  high  school. 

For  descriptive  purposes  subjects  were  divided  into  six  ethnic  groups;  Caucasian,  Afro- 
American,  Asian,  Hispanic,  North  American  Indian,  and  other.  The  categories  are  a  combination 
of  the  population  and  ethnic  group  codes  taken  from  enlistment  records.  Caucasian  was  defined 
by  the  population  code  Caucasian  (C)  and  the  ethnic  code  none  (Y).  Afro-American  was  defined 
by  the  population  code  Negroid/African/Black  (N)  unless  a  Hispanic  ethnic  code  was  also 
checked  (then  the  person  would  be  defined  as  Hispanic,  see  below).  The  Asian  group  was 
defined  by  the  population  code  Asian/Mongoloid/Yellow  (M)  and/or  ethnic  codes  for  other  Asian 
descent  (3),  Filipino  (5),  Chinese  (G),  Japanese  (J),  Korean  (K),  Vietnamese  (V),  Melanesian  (E), 
Micronesian  (W),  Polynesian  (L),  and  other  Pacific  Island  descent  (Q).  Regardless  of  the 
population  code,  the  Hispanic  group  was  defined  by  ethnic  codes  for  other  Hispanic  descent  (1), 
Puerto  Rican  (4),  Mexican  (6),  Cuban  (9),  and  Latin  American  with  Hispanic  descent  (S).  The 
North  American  Indian  group  was  defined  by  the  population  code  for  American  Indian/Red  (R) 
and  by  ethnic  codes  for  U.S./Canadian  Indian  Tribes  (2),  Eskimo  (7),  and  Aleut  (8).  A  final 
group  labeled  Other  was  created  from  the  population  code  Other  (X)  and  the  ethnic  codes  Other 
(X)  (unless  Caucasian),  Indian  (from  India;  D),  and  Unknown  (Z).  The  distribution  of  subjects 


The  contributions  of  the  TASP  to  the  planning  and  design  of  this  project  are  greatly  appreciated.  The  panel  was  chaired  by  Dr.  Bruce  Bloxom,  who 
provided  the  overall  framework  by  which  the  proposed  tests  were  to  be  evaluated,  provided  leadership,  organization,  and  technical  quidance,  and 
documented  the  deliberations  of  the  panel  in  the  minutes  of  the  meetings..  Clint  Walker,  of  the  Aimy  Research  Institute,.  Lonnie  Valentine,  from  the 
Air  Force  Human  Resources  Laboratory  ,  and  John  Wolfe,  from  the  Navy  Personnel  Research  and  Development  Center,  provided  detailed  proposals 
for  tests  to  be  considered  by  the  panel.  The  panel  also  proposed  and  decided  on  the  particular  samples  that  were  to  be  collected,  and  established 
contacts  for  arranging  the  testing  of  research  subjects. 
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across  the  six  ethnic  groups  was  71.1%  Caucasian,  16.5%  Afro-American,  5.9%  Hispanic,  2.2% 
Asian,  0.8%  North  American  Indian,  and  3.4%  Other/Unknown. 

Enhanced  Computer  Administered  Test  Content 

The  goal  of  EC  AT  was  to  broaden  the  ASVAB.  Table  1  shows  the  10  tests  that  comprise  the 
ASVAB.  These  tests  represent  Verbal  Ability,  Mathematical  Ability,  Technical  Knowledge,  and 
Perceptual  Speed.  Across  Services,  the  ASVAB's  four  factor  structure  was  the  focal  or  starting 
point  for  new  predictor  research.  Specifically,  the  assumption  was  made  that  the  scope  of  human 
intellectual  and  nonintellectual  skills  was  much  greater  than  that  represented  by  the  ASVAB,  and 
that  capturing  this  breadth  held  the  greatest  promise  for  improving  personnel  selection  and/or 
classification. 


Table  1 


Tests  in  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) 


Construct 

Test 

Description 

Verbal 

Paragraph  Comprehension  (PC) 

A  15-item  reading  comprehension  test 

Ability 

Word  Knowledge  (WK) 

A  35-item  vocabulary  test  using  words  embedded  in 
sentences  or  synonyms 

General  Science  (GS) 

A  25-item  knowledge  test  of  physical  and  biological 
sciences 

Math  Ability 

Arithmetic  Reasoning  (AR) 

A  30-item  arithmetic  word  problem  test 

Math  Knowledge  (MK) 

A  25-item  test  of  algebra,  geometry,  fractions,  decimals,  and 
exponents 

Technical 

Mechanical  Comprehension  (MC) 

A  25-item  test  of  mechanical  and  physical  principles 

Knowledge 

Auto  and  Shop  Information  (AS) 

A  25-item  knowledge  test  of  automobiles,  shop  practices, 
tools,  and  tool  use 

Electronic  Information  (El) 

A  20-item  test  about  electronics,  radio,  and  electrical 
principles  and  information 

Clerical  Skills 

Numerical  Operations  (NO) 

A  50-item  speeded  addition,  subtraction,  multiplication,  and 
division  test  using  one  and  two  digit  numbers 

Coding  Speed  (CS) 

An  84-item  speeded  test  requiring  the  recognition  of 
number  strings  arbitrarily  associated  with  words  in  a  table 

Table  2  shows  the  9  tests  that  comprise  the  ECAT  battery  including  a  brief  description  of 
each  test.  The  battery  requires  a  maximum  of  3  hours  with  most  individuals  finishing  in  under 
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2  hours.  The  tests  are  grouped  by  the  aptitude  construct  they  were  designed  to  measure: 
Nonverbal  Reasoning,  Spatial  Ability,  Psychomotor  Skill,  and  Perceptual  Speed.  The  tests  were 
administered  on  Hewlett-Packard  portable  IPCs,  which  are  the  delivery  systems  for  the 
Computerized  Adaptive  Testing  version  of  the  ASVAB  (CAT-ASVAB).  The  keyboard  was 
modified  by  using  a  plastic  mask  that  revealed  only  the  designated  response  keys  along  with  a  key 
labeled  HELP  that  could  be  pressed  during  testing  to  suspend  the  program  and  request  assistance. 
The  S,  F,  H,  K,  and  ;  keys  were  relabeled:  A,  B,  C,  D,  and  E.  The  space  bar  was  relabeled 
ENTER.  The  numeric  keypad  keys  retained  their  meanings.  The  last  three  ECAT  tests  used  a 
custom-made  input  device  referred  to  as  a  response  pedestal.  The  response  pedestal  has  color- 
coded  buttons,  two  slide-potentiometers,  and  two  joy-sticks  which  are  used  to  respond  to  items 
(see  Figure  1).  In  addition,  the  response  pedestal  contains  a  key  labeled  HELP  that  behaved  like 
the  corresponding  key  on  the  keyboard. 


Table  2 

Tests  in  the  Joint-Service  ECAT  Battery 


Construct 

Test 

Description 

Nonverbal  Reasoning 

Mental  Counters  (CT) 

A  40-item  Working  Memory  test  using  figural  content; 
a  nonverbal  reasoning  test 

Sequential  Memory  (SM) 

A  35-item  Working  Memory  test  using  numerical 
content;  a  nonverbal  reasoning  test 

Figural  Reasoning  (FR) 

A  35-item  series  extrapolation  test  using  figural 
content;  a  nonverbal  reasoning  test 

Spatial  Ability 

Integrating  Details  (ID) 

A  40-item  spatial  problem  solving  test 

Assembling  Objects  (AO) 

A  32-item  spatial  and  semi-mechanical  test 

Spatial  Orientation  (SO) 

A  24-item  spatial  apperception/rotation  test 

Psychomotor  Skill 

One-Hand  Tracking  (Tl) 

An  18-item  single  limb  psychomotor  tracking  test 

Two-Hand  Tracking  (T2) 

An  18-item  multi-limb  psychomotor  tracking  test 

Perceptual  Speed 

Target  Identification  (TI) 

A  36-item  reaction  time-based  figural  perceptual  speed 
test 

Note.  ECAT  =  Enhanced  Computer  Administered  Testing. 
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Top  View 


Side  View 


Figure  1.  Response  Pedestal  (two  views). 
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Since  most  of  the  ECAT  tests  are  quite  novel,  a  brief  description  of  each  test  is  warranted. 
Each  test  will  be  illustrated  with  a  sample  item  accompanied  by  an  abstract  of  the  actual 
instructions,  which  often  require  five  to  ten  screens,  some  with  animation. 


Nonverbal  Reasoning  Tests 

Mental  Counters  (CT)--is  a  complex  40  item  working  memory  test.  See  Figure  2  for  an 
example.  Each  screen  contains  three  horizontal  lines,  arrayed  left  to  right.  Each  line  represents  a 
counter  with  an  initial  value  of  zero.  During  an  item,  boxes  appear  sequentially,  one  at  a  time, 
either  above  or  below  one  of  the  three  lines.  If  a  box  appears  above  a  line,  the  value  for  that 
counter  is  incremented  by  +1.  If  a  box  appears  below  a  line,  that  counter  is  decremented  by  1. 
On  each  trial  either  five  or  seven  boxes  appear.  The  boxes  appear  at  one  of  two  rates,  either  one 
every  1.33  seconds  or  one  every  .75  seconds.  The  task  is  to  make  a  series  of  rapid  calculations 
and  to  select,  from  a  four-alternative  multiple  choice  menu,  the  set  of  correct  final  counter  values. 
Number  of  correct  responses  is  the  summary  score. 


3 

1 

2,5 

4 

Three  independent  counters  (center  horizontal  lines)  begin  with  starting  values  of  0.  Boxes  are  sequentially 
displayed,  then  removed,  in  the  order  shown.  If  a  box  appears  above  a  line  the  counter  is  incremented  by  1,  if 
below  the  line,  it  is  decremented  by  1.  The  final  counter  values  for  this  item  would  be  (in  order)  -2,  +1, 0. 

Figure  2.  Mental  Counters  test. 
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Sequential  Memory  (SM)--is  another  complex  test  of  working  memory.  See  Figure  3  for 
an  example.  Each  item  consists  of  three  to  five  horizontally  arrayed  dots  on  the  screen.  Each  dot 
is  given  a  numerical  value;  these  must  be  memorized.  The  item  is  then  presented  in  a  series  of  5 
to  7  "calls"  to  the  dots;  where  each  call  is  announced  by  briefly  turning  one  of  the  dots  into  an 
"X."  The  person  must  report  the  digit  string  that  corresponds  to  the  order  that  the  dots  were 
"called."  In  the  second  half  of  the  test,  after  all  the  calls  for  an  item  have  been  made,  the  examinee 
is  told  to  translate  each  number  in  the  ordered  number  list  into  a  different  number  and  then  type  in 
the  new  ordered  list.  There  are  10  items  in  the  first  half  of  the  test  and  25  in  the  second  half  of 
the  test.  The  dependent  variable  is  the  proportion  of  digits  correct. 
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The  start  values  indicate  the  numbers  assigned  to  each  position.  Following  this,  each  time  an  X  appears,  it  "calls" 
the  corresponding  number.  When  the  X  appears  in  the  center  position,  the  2  is  called.  When  the  X  appears  in  the 
left  position,  the  5  is  called.  When  the  X  appears  in  the  right  position,  the  8  is  called.  Remember  the  sequence  erf 
calls.  (Answer:  2,  8, 2, 5,  5) 


Figure  3.  Sequential  Memory  test. 
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Figural  Reasoning  (FR)--is  a  figural  inductive  reasoning  (or  series  extrapolation)  test.  See 
Figure  4  for  an  example.  Items  use  a  combination  of  geometric  forms  and  arbitrary  figures 
presented  in  a  series  of  four  frames.  The  task  is  to  induce  the  transformation  rule  controlling  the 
series  and  then  select  one  of  five  alternatives  that  correctly  completes  the  series.  The  dependent 
variable  is  number  correct  of  30  items. 


Figure  Series 


A  B  C  D 


Which  alternative  shows  the  next  frame  in  the  figure  series?  (Answer:  D) 


Figure  4.  Figural  Reasoning  test. 
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Spatial  Ability  Tests 

Integrating  Details  (ID)--is  a  complex  40  item  spatial  problem  solving  test.  See  Figure  5 
for  an  example.  Each  item  consists  of  two  separate  screens.  The  first  screen  contains  from  two 
to  six  regular  geometric  puzzle  pieces  that  must  be  mentally  fused  to  form  a  complete  object. 
This  is  much  like  a  jig-saw  puzzle.  Having  connected  all  of  the  puzzle  pieces,  the  individual  must 
remember  the  final  object,  then  press  a  response  key.  The  puzzle  pieces  are  replaced  by  a  new 
screen  with  a  single  completed  object.  The  task  is  to  indicate  if  the  displayed  object  is  the  product 
of  the  original  puzzle  pieces.  Accuracy  is  the  dependent  measure. 


Screen  1 


Screen  2 


The  top  frame  is  presented  and  the  examinee  has  as  long  as  necessary  to  mentally  construct  a  complete  object 
Following  a  key  press,  the  bottom  frame  is  presented.  The  subject  has  as  long  as  necessary  to  decide  if  the  puzzle 
pieces  would  have  constructed  this  object.  Toggling  between  screens  is  not  allowed.  (Answer:  Same.) 

Figure  5.  Integrating  Details  test. 
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Assembling  Objects  (AO)— is  a  spatial  construction  test.  See  Figure  6  for  an  example. 
Each  item  consists  of  a  frame  with  several  (2-6)  separate  elements.  The  task  is  to  choose  from 
four  alternatives  the  answer  that  correctly  represents  how  the  elements  should  be  connected. 
There  are  32  items  in  the  test.  The  first  15  items  are  semi-mechanical  items  with  labels  indicating 
how  the  elements  should  be  connected.  The  final  17  items  consist  of  a  disheveled  jig-saw  and 
four  complete  ones;  the  task  is  to  chose  the  correct  alternative.  The  dependent  variable  is  number 
correct. 


Figure  6.  Assembling  Objects  item  types. 
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Spatial  Orientation  (SO)--is  a  spatial  perspective  test.  See  Figure  7  for  an  example.  Each  item 
consists  of  an  environmental  view,  such  as  a  bridge  over  a  river  or  a  house  with  an  apparent  horizon. 
These  views  are  rotated  away  from  the  "natural"  horizon.  At  the  bottom  of  the  frame  is  a  circle  with  a  dot 
on  the  perimeter.  The  task  is  to  rotate  the  frame  around  the  view  until  it  corresponds  with  the  natural 
horizon  and  determine  where  the  dot  on  the  circle  would  be  located.  This  information  is  used  to  select 
which  of  five  alternatives  correctly  shows  the  dot  following  rotation.  The  dependent  variable  is  the  number 
of  items  correct. 


The  sample  problem  contains  a  picture  with  a  frame  around  it.  The  bottom  of  the  frame  has  a  circle  with  a  dot 
inside,  carved  into  the  frame.  Imagine  that  only  the  frame  can  be  turned,  and  the  picture  inside  can  not  be  moved. 
Then,  to  match  up  the  bottom  of  the  frame  with  the  bottom  of  the  picture,  you  would  turn  the  frame  until  the  circle 
with  a  dot  is  located  at  the  bottom  of  the  picture.  Your  task  is  to  figure  out  exactly  what  the  circle  with  the  dot 
inside  will  look  like  in  its  new  position,  after  the  frame  has  been  turned.  (Answer:  E) 

Figure  7.  Spatial  Orientation  test. 
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Psychomotor  Skill 

One-Hand  Tracking  (Tl)-is  a  psychomotor  test  that  uses  the  response  pedestal.  See 
Figure  8  for  an  example.  Each  item  begins  with  a  "path"  on  the  computer  screen.  The  path  is  a 
contiguous  string  that  goes  up/down  and/or  right/left,  parallel  with  the  sides  of  the  screen,  making 
only  90  degree  turns.  At  one  end  of  the  path  is  a  diamond  indicating  the  path’s  termination  point. 
Starting  at  the  other  end  is  a  box  that  travels  forward  along  the  path.  The  subject  moves  a  joy¬ 
stick  that  controls  the  movement  of  a  "cross-hair."  The  task  is  to  keep  the  cross-hair  on  the 
moving  box.  Items  vary  in  terms  of  the  length  of  the  path  which  is  inversely  related  to  the  speed 
at  which  the  box  moves  (total  item  duration  is  thus  constant).  For  each  item,  the  "score"  is  the 
average  absolute  Cartesian  pixel  distance  between  the  cross-hair  and  the  moving  box  (a  distance 
reading  is  taken  every  50  ms  during  the  item).  The  dependent  variable  is  the  average  distance- 
off-target  across  18  items. 


Keep  the  "cross  hair"  on  the  target  (square)  until  the  target  movement  stops  (at  the  diamond). 

Figure  8.  One-Hand  and  Two-Hand  Tracking  items. 
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Two-Hand  Tracking  (T2)-is  another  psychomotor  test  that  has  exactly  the  same  structure 
and  task  constraints  as  One-Hand  Tracking  (see  Figure  8).  The  difference  is  that  cross-hair 
movement  is  controlled  by  two  slide  potentiometers:  one  slide  controls  horizontal  (left/right) 
movement  while  the  other  controls  vertical  (up/down)  motion  (see  Figure  1).  One  hand  must  be 
used  for  each  slide  control.  Number  of  items,  scoring,  and  final  score  are  the  same  as  One-Hand 
Tracking. 


Perceptual  Speed 

Target  Identification  (Tl)-is  a  hybrid  test  combining  aspects  of  choice  reaction  time  and 
spatial  mental  rotation  tests.  See  Figure  9  for  an  example.  Each  item  consists  of  a  target  figure  in 
the  top  half  of  the  screen  and  three  alternative  figures  in  the  bottom  half.  The  figures  are 
schematic  line  drawings  of  simple  objects,  such  as  trucks,  helicopters,  and  tanks.  The  target  may 
be  rotated,  distorted  (e.g.,  shrunken),  or  both,  but  the  correct  alternative  will  be  in  a  “natural” 
upright  position.  The  task  is  to  select  the  correct  alternative  as  rapidly  as  possible.  Before  each 
item  examinees  must  simultaneously  press  four  “Home”  buttons,  two  on  the  left  and  two  on  the 
right  side  of  the  response  pedestal,  essentially  pinning  their  hands  (see  Figure  1).  As  soon  as  the 
examinee  decides  upon  an  answer,  either  hand  may  be  used  to  press  the  button  corresponding  to 
the  selected  alternative.  The  dependent  variable  is  the  average  correct  decision  time  across  the  36 
items,  where  decision  time  is  defined  as  the  time  between  item  presentation  and  button  release. 


Target 


Blue 


Yellow 


White 


While  keeping  fingers  on  Home  keys,  determine  which  object  matches  the  Target,  then  press  the  correctly  colored 
key.  (Answer:  White) 


Figure  9.  Target  Identification  test. 


15 


Six  of  the  ECAT  tests  (i.e.,  FR,  AO,  SO,  Tl,  T2,  and  TI)  were  developed  by  the  Army 
Research  Institute  as  part  of  Project  A  (Peterson,  Hough,  Dunnette,  Rosse,  Houston,  &  Toquam, 
1990).  The  remaining  three  tests  (CT,  SM,  and  ID)  were  developed  by  the  Navy  Personnel 
Research  and  Development  Center. 

Descriptive  Statistics 

Table  3  contains  simple  descriptive  statistics  for  the  ASVAB  and  ECAT  tests.  The  sample 
was  lightly  edited  for  unmotivated  and  extremely  poor  performing  individuals  before  analyses 
were  conducted.  The  first  entry  in  Table  3,  the  Armed  Services  Qualification  Test  (AFQT),  is,  a 
composite  of  the  mathematical  (Arithmetic  Reasoning  and  Mathematical  Knowledge)  and  verbal 
(Word  Knowledge  and  Paragraph  Comprehension)  tests  and  is  used  to  determine  eligibility  for 
military  service.  The  AFQT  is  also  a  good  estimate  of  general  intellectual  performance.  The 
AFQT  composite  is  expressed  as  a  cumulative  percentile  with  a  mean  of  50  and  a  standard 
deviation  of  28.7  in  a  nationally  representative  sample  of  service  eligible  individuals.  The  current 
sample  had  a  mean  AFQT  of  60.3  and  a  standard  deviation  of  18.0.  Thus,  the  sample  was  .36 
standard  deviation  units  above  the  mean  resulting  in  a  37%  reduction  in  the  standard  deviation 
due  to  explicit  selection  (truncating  the  lower  tail  of  the  distribution).  The  individual  ASVAB 
tests  are  all  scaled  to  a  mean  of  50  with  a  standard  deviation  of  10;  they  all  show  elevated  means 
and  truncated  standard  deviations  due  to  selection  effects.  Little  can  be  said  about  the  ECAT 
tests  except  that  they  all  have  reasonable  distributional  properties  and  that  the  means  and  standard 
deviations  reported  here  are  similar  to  those  found  in  other  samples  where  the  tests  were  used. 

The  last  column  of  Table  3  contains  uncorrected  test  reliability  estimates.  The  ASVAB 
reliabilities  are  based  on  a  4  week  retest  period  using  alternate  forms;  the  median  reliability  is  .74 
(Moreno  &  Segall,  1992).  As  described  in  Appendix  E,  the  ECAT  reliabilities  were  computed  on 
a  five  week  delayed  readministration  of  the  same  forms;  the  median  reliability  is  .80  (Larson  & 
Alderton,  1992a,  1992b) 

Table  4  shows  the  means,  standard  deviations,  and  correlations  of  the  ASVAB  and  ECAT 
tests  after  correction  for  multivariate  range  restriction  using  the  10  ASVAB  variables  as 
explicitly  selected  variables  with  population  covariances  equal  to  those  of  the  1991  joint-services 
applicant  population  (N  =  650,278). 
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Table  3 


Descriptive  Statistics  for  the  ASVAB  and  EC  AT  Test  Batteries 


Test 

Mean 

Standard 

Deviation 

Minimum 

Maximum 

Retest 

Reliability 

Armed  Forces  Qualification  Test  (AFQT) 

61.179 

17.917 

17.000 

99.000 

General  Science  (GS) 

53.255 

7.419 

23.000 

69.000 

.73 

Arithmetic  Reasoning  (AR) 

53.610 

6.905 

31.000 

66.000 

.77 

Word  Knowledge  (WK) 

53.044 

5.354 

20.000 

61.000 

.82 

Paragraph  Comprehension  (PC) 

53.229 

5.740 

20.000 

62.000 

.48 

Numerical  Operations  (NO) 

54.208 

6.583 

20.000 

62.000 

.71 

Coding  Speed  (CS) 

53.248 

6.937 

22.000 

72.000 

.75 

Auto-Shop  Information  (AS) 

53.614 

8.051 

28.000 

69.000 

.77 

Math  Knowledge  (MK) 

55.125 

6.876 

30.000 

68.000 

.82 

Mechanical  Comprehension  (MC) 

54.958 

7.703 

27.000 

70.000 

.70 

Electronics  Information  (El) 

52.593 

7.945 

23.000 

70.000 

.65 

Mental  Counters  (CT) 

0.724 

0.175 

0.200 

1.000 

.79 

Sequential  Memory  (SM) 

0.688 

0.134 

0.160 

1.000 

.81 

Figural  Reasoning  (FR) 

0.669 

0.188 

0.100 

1.000 

.75 

Integrating  Details  (ID) 

0.760 

0.127 

0.375 

1.000 

.79 

Assembling  Objects  (AO) 

0.629 

0.193 

0.094 

1.000 

.83 

Spatial  Orientation  (SO) 

0.517 

0.247 

0.125 

1.000 

.75 

One-Hand  Tracking  (Tl) 

2765.374 

391.724 

2003.611 

4867.111 

.84 

Two-Hand  Tracking  (T2) 

3639.163 

471.978 

2391.278 

5460.722 

.91 

Target  Identification  (TI) 

1.835 

0.604 

0.280 

5.610 

.80 

Note.  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery,  EC  AT  =  Enhanced  Computer  Administered  Testing. 
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Table  4 


Range-Corrected  Means,  Standard  Deviations,  and  Intercorrelations  of 
ASVAB  and  ECAT  Tests  (N  =  10,963) 


Test 

GS 

AR 

WK 

PC 

NO 

CS 

AS 

MK 

MC 

El 

Mean  50.6150 

Std.Dev.  8.7726 

50.6636 

8.6454 

51.3114 

7.3541 

51.1558 

7.9640 

52.5122 

8.0131 

52.2662 

7.8118 

51.4087 

9.1677 

51.2103 

8.6890 

51.9408 

9.1272 

50.3326 

8.8559 

GS 

1.0000 

0.6111 

0.7201 

0.6079 

0.2751 

0.2487 

0.5202 

0.5542 

0.6377 

0.6245 

AR 

0.6111 

1.0000 

0.5963 

0.5743 

0.4703 

0.3953 

0.4004 

0.7069 

0.6134 

0.4868 

WK 

0.7201 

0.5963 

1.0000 

0.7316 

0.3244 

0.3278 

0.4366 

0.4968 

0.5473 

0.5344 

PC 

0.6079 

0.5743 

0.7316 

1.0000 

0.3959 

0.3859 

0.3391 

0.4997 

0.4852 

0.4445 

NO 

0.2751 

0.4703 

0.3244 

0.3959 

1.0000 

0.6401 

0.0470 

0.4961 

0.2279 

0.1452 

CS 

0.2487 

0.3953 

0.3278 

0.3859 

0.6401 

1.0000 

0.0583 

0.4078 

0.2212 

0.1471 

AS 

0.5202 

0.4004 

0.4366 

0.3391 

0.0470 

0.0583 

1.0000 

0.1966 

0.6181 

0.6692 

MK 

0.5542 

0.7069 

0.4968 

0.4997 

0.4961 

0.4078 

0.1966 

1.0000 

0.4939 

0.3696 

MC 

0.6377 

0.6134 

0.5473 

0.4852 

0.2279 

0.2212 

0.6181 

0.4939 

1.0000 

0.6304 

El 

0.6245 

0.4868 

0.5344 

0.4445 

0.1452 

0.1471 

0.6692 

0.3696 

0.6304 

1.0000 

CT 

0.3684 

0.5582 

0.3409 

0.3529 

0.3705 

0.3490 

0.2093 

0.5163 

0.4259 

0.2685 

SM 

0.3606 

0.5318 

0.3682 

0.3704 

0.3412 

0.3387 

0.1703 

0.4892 

0.3854 

0.2373 

FR 

0.5026 

0.5945 

0.4727 

0.4425 

0.3073 

0.2872 

0.3108 

0.5457 

0.5313 

0.3914 

ID 

0.5024 

0.5695 

0.4310 

0.3909 

0.2601 

0.2584 

0.3787 

0.5174 

0.5743 

0.4315 

AO 

0.4743 

0.5142 

0.3990 

0.3611 

0.2371 

0.2669 

0.3889 

0.4675 

0.5559 

0.4254 

SO 

0.4888 

0.5366 

0.4392 

0.3930 

0.2250 

0.2380 

0.3955 

0.4824 

0.5622 

0.4291 

T1 

-0.2882 

-0.2956 

-0.2440 

-0.2272 

-0.2008 

-0.1967 

-0.2589 

-0.2608 

-0.3677 

-0.2659 

T2 

'  -0.3405 

-0.3369 

-0.2967 

-0.2614 

-0.1910 

-0.2104 

-0.3230 

-0.2806 

-0.4362 

-0.3233 

TI 

-0.3151 

-0.2651 

-0.2537 

-0.2224 

-0.1781 

-0.1917 

-0.2274 

-0.2300 

-0.3216 

-0.2349 

Test 

CT 

SM 

FR 

ID 

AO 

SO 

Tl 

T2 

TI 

Mean 

0.7209 

0.4447 

2827.3 

3722.98 

Std.Dev. 

IW 

0.1404 

0.2726 

406.0732 

494.5723 

GS 

0.3684 

0.3606 

0.5026 

0.5024 

0.4743 

0.4888 

-0.2882 

-0.3405 

-0.3151 

AR 

0.5582 

0.5318 

0.5945 

0.5695 

0.5142 

0.5366 

-0.2956 

-0.3369 

-0.2651 

WK 

0.3409 

0.3682 

0.4727 

0.4310 

0.3990 

0.4392 

-0.2440 

-0.2967 

-0.2537 

PC 

0.3529 

0.3704 

0.4425 

0.3909 

0.3611 

0.3930 

-0.2272 

-0.2614 

-0.2224 

NO 

0.3705 

0.3412 

0.3073 

0.2601 

0.2371 

0.2250 

-0.2008 

-0.1910 

-0.1781 

CS 

0.3490 

0.3387 

0.2872 

0.2584 

0.2669 

0.2380 

-0.1967 

-0.2104 

-0.1917 

AS 

0.2093 

0.1703 

0.3108 

0.3787 

0.3889 

0.3955 

-0.2589 

-0.3230 

-0.2274 

MK 

0.5163 

0.4892 

0.5457 

0.5174 

0.4675 

0.4824 

-0.2608 

-0.2806 

-0.2300 

MC 

0.4259 

0.3854 

0.5313 

0.5743 

0.5559 

0.5622 

-0.3677 

-0.4362 

-0.3216 

El 

0.2685 

0.2373 

0.3914 

0.4315 

0.4254 

0.4291 

-0.2659 

-0.3233 

-0.2349 

CT 

1.0000 

0.6288 

0.5586 

0.5530 

0.5700 

0.5067 

-0.3787 

-0.3889 

-0.2964 

SM 

0.6288 

1.0000 

0.5422 

0.4939 

0.4885 

0.4583 

-0.3162 

-0.3343 

-0.2807 

FR 

0.5586 

0.5422 

1.0000 

0.5930 

0.5768 

0.5431 

-0.3464 

-0.3713 

-0.2939 

ID 

0.5530 

0.4939 

0.5930 

1.0000 

0.6461 

0.5736 

-0.3808 

-0.4061 

-0.3287 

AO 

0.5700 

0.4885 

0.5768 

0.6461 

1.0000 

0.5779 

-0.3801 

-0.4276 

-0.3664 

SO 

0.5067 

0.4584 

0.5431 

0.5736 

0.5779 

1.0000 

-0.3668 

-0.4084 

-0.2815 

Tl 

-0.3787 

-0.3162 

-0.3464 

-0.3808 

-0.3801 

-0.3668 

1.0000 

0.7522 

0.3631 

T2 

-0.3889 

-0.3343 

-0.3713 

-0.4061 

-0.4276 

-0.4084 

0.7522 

1.0000 

0.3844 

TI 

-0.2964 

-0.2807 

-0.2939 

-0.3287 

-0.3664 

-0.2815 

0.3631 

0.3844 

1.0000 

Note.  ASVAB  =  Aimed  Services  Vocational  Aptitude  Battery,  ECAT  =  Enhanced  Computer  Administered  Testing.  See  Tables  1  &  2  for  test  names. 
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Adverse  Impact 

From  inspection  of  the  content  of  the  sample  items,  one  can  conclude  that  the  ECAT  tests  are 
relatively  knowledge-free  as  compared  with  the  ASVAB  (i.e.,  they  do  not  require  knowledge  acquired 
through  formal  education).  They  may  be  described  as  tests  of  fluid  intelligence,  rather  than  the  crystallized 
intelligence  measured  by  the  ASVAB.  Table  5  demonstrates  this  empirically  by  correlating  the  ASVAB  and 
ECAT  tests  with  Years  of  Education.  With  the  exception  of  Auto-Shop  Information,  the  ASVAB 
correlations  with  education  are  generally  larger  than  the  ECAT  correlations.  The  correlations  were  not 
corrected  for  range-restriction,  but  such  corrections  should  increase  the  correlation  with  education  more 
for  the  ASVAB  tests,  which  are  explicitly  used  for  selection,  than  for  the  ECAT  tests.  Lower  correlations 
of  the  ECAT  tests  with  education  should  cause  the  ECAT  tests  to  have  less  adverse  impact  on 
educationally  disadvantaged  subgroups. 


Table  5 

Correlations  of  ASVAB  and  ECAT  Tests  With  Years  of  Education  (N  =  10756) 


Test 

Correlation  with 

Years  of  Education 

Armed  Forces  Qualification  Test  (AFQT) 

.166* 

General  Science  (GS) 

* 

m 

00 

o 

Arithmetic  Reasoning  (AR) 

.107* 

Word  Knowledge  (WK) 

.109  * 

Paragraph  Comprehension  (PC) 

.078* 

Numerical  Operations  (NO) 

.111  * 

Coding  Speed  (CS) 

.116* 

Auto-Shop  Information  (AS) 

.011  ns 

Math  Knowledge  (MK) 

.179* 

Mechanical  Comprehension  (MC) 

.058* 

Electronics  Information  (El) 

.064* 

Mental  Counters  (CT) 

.058* 

Sequential  memory 

.049* 

Integrating  Details  (ID) 

.055  * 

Assembling  Objects  (AO) 

.039  * 

Spatial  Orientation  (SO) 

.061  * 

Figural  Reasoning  (FR) 

.065  * 

One-hand  Tracking 

.036* 

Two-hand  Tracking 

.014  ns 

Target  Identification  (Tl) 

.014  ns 

Note.  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery,  ECAT  =  Enhanced  Computer  Administered  Testing. 
*  p  <  .01 .  ns  =  not  significant. 
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Table  6  confirms  this  hypothesis.  It  shows  the  differences  in  mean  test  scores  between 
Caucasians,  and  Afro-Americans,  Asians,  and  Hispanics  (See  Appendix  F  for  further  details). 
The  four  tests  with  the  largest  adverse  impact  are  all  ASVAB  tests  -  GS,  WK,  AS,  and  MC.  The 
subgroups  differ  on  which  tests  have  the  least  adverse  impact,  but  the  ECAT  tests  compare 
favorably  with  the  ASVAB  tests.  Since  the  sample  was  explicitly  selected  by  ASVAB  scores, 
correction  for  range  restriction  should  increase  the  adverse  impact  of  ASVAB  tests  more  than  of 
ECAT  tests. 


Table  6 


Subgroup  Differences  in  ASVAB  and  ECAT  Test  Means 


Caucasian  - 

Caucasian  - 

Caucasian  - 

Afro-American 

Asian 

Hispanic 

Variable 

z 

Z 

Z 

Years  of  Education 

-.058  * 

-.288  ** 

.133  ** 

Educational  Level 

.030 

.265  ** 

-.146  ** 

Language 

.006 

-1.988  ** 

-.234  ** 

AFQT 

.736  ** 

.302  ** 

.370  ** 

General  Science  (GS) 

.818  ** 

.609  ** 

.475  ** 

Arithmetic  Reasoning  (AR) 

.753  ** 

.187  ** 

293  ** 

Word  Knowledge  (WK) 

.736  ** 

.755  ** 

.532  ** 

Paragraph  Comprehension  (PC) 

.515  ** 

.375  ** 

219  ** 

Numerical  Operations  (NO) 

.023 

-.189  ** 

.022 

Coding  Speed  (CS) 

142  ** 

-.073 

.051 

Auto-Shop  Information  (AS) 

1.106** 

.829  ** 

.638  ** 

Math  Knowledge  (MK) 

.164** 

-.396  ** 

-.017 

Mechanical  Comprehension  (MC) 

90i  ** 

.430  ** 

.440  ** 

Electronics  Information  (El) 

719  ** 

.358  ** 

.344  ** 

Mental  Counters  (CT) 

.656  ** 

-.100 

.089* 

Sequential  Memory  (SM) 

.445  ** 

.139* 

.248  ** 

Integrating  Details  (ID) 

729  ** 

-.023 

.116** 

Assembling  Objects  (AO) 

.713  ** 

.010 

.097* 

Spatial  Orientation  (SO) 

.694  ** 

.165* 

.169  ** 

Figural  Reasoning  (FR) 

.546  ** 

.103 

.196  ** 

One-hand  Tracking 

-.565  ** 

-.292  ** 

-.026 

Two-hand  Tracking 

-.701  ** 

-.314  ** 

-.113** 

Target  Identification  (TI) 

-.485  ** 

-.400  ** 

-.179  ** 

Note.  ASVAB  =  Aimed  Services  Vocational  Aptitude  Battery.  ECAT  =  Enhanced  Computer  Administered  Testing.  Z  values  are  differences  in  ECAT 
sample  means  divided  by  the  Caucasian  group  standard  deviations. 

*  p .  .05  ** p<  .01. 
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Factor  Analysis 

One  of  the  goals  in  selecting  tests  for  inclusion  in  the  ECAT  battery  was  to  expand  upon  the 
domain  of  abilities  measured  by  the  ASVAB.  To  determine  how  successful  the  efforts  were, 
factor  analyses  were  conducted  to  determine:  (1)  the  underlying  dimensions  in  each  test  battery, 
(2)  the  overlap  in  measuring  general  intelligence  across  the  batteries,  and  (3)  the  factor  structure 
when  both  batteries  are  combined.  A  number  of  factor  analytic  solutions  were  obtained  by 
varying  factor  extraction  methods,  number  of  factors  extracted,  method  of  rotation,  and  initial 
communality  estimates.  Only  the  hierarchical  solutions  will  be  described.  For  hierarchical 
analyses,  a  Promax  rotation  was  used  at  the  primary  level(s)  with  the  entire  hierarchical  solution 
orthogonalized  using  the  Schmid-Leiman  technique  (Schmid  &  Leiman,  1957).  In  the  Schmid- 
Leiman  orthogonalization,  the  effects  of  the  second-order  (g)  loadings  are  removed  from  the  first- 
order  loadings.  These  residuals  of  the  first-order  loadings  will  be  reported,  rather  than  the  original 
first-order  loadings,  which  are  much  larger.  Therefore,  residual  loadings  as  small  as  .20  can  be 
reported  as  useful,  in  contrast  to  the  .40  standard  that  is  commonly  used  for  interpreting  primary 
rotated  solutions. 

All  analyses  are  based  on  the  corrected  correlations  reported  in  Table  4.  Appendix  B  contains 
the  uncorrected  and  corrected  correlations  with  other  measures,  as  well..  The  primary  factor 
loadings  before  orthogonalization  are  reported  in  Appendix  C. 


Armed  Services  Vocational  Aptitude  Battery  Factor  Structure 

Table  7  reports  the  results  of  the  hierarchical  solution  for  the  ASVAB.  In  this  and  the  next 
two  tables,  where  the  tests  had  Promax  loadings  greater  than  .40,  the  hierarchical  residual 
loadings  appear  in  bold-face  type.  The  four  primary  factors  are:  (1)  Technical  Knowledge 
defined  by  Auto-Shop  Information,  Mechanical  Comprehension,  and  Electronics  Information; 
(2)  Verbal  Ability  defined  by  Word  Knowledge  and  Paragraph  Comprehension;  (3)  Clerical  Speed 
defined  by  Numerical  Operations  and  Coding  Speed;  and  (4)  Mathematical  Ability  defined  by 
Arithmetic  Reasoning  and  Math  Knowledge.  The  only  factorially  complex  test  is  General 
Science,  which  splits  its  specific  variance  across  the  Technical  and  Verbal  factors.  This  factor 
structure  is  routinely  found  to  describe  the  ASVAB’s  intercorrelations.  All  of  the  tests  load  on  the 
hierarchical  general  ability  measure  (g)  which  accounted  for  40%  of  the  intercorrelational 
variance. 
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Table  7 


Orthogonalized  Hierarchical  Factor  Solution  for  the  ASVAB 


Test 

g 

Technical 

Verbal 

Clerical 

Math 

General  Science  (GS) 

.738 

.245 

.241 

-.073 

Arithmetic  Reasoning  (AR) 

.753 

.164 

.054 

.110 

Word  Knowledge  (WK) 

.782 

.024 

.504 

-.009 

Paragraph  Comprehension  (PC) 

.703 

.019 

.360 

.124 

Numerical  Operations  (NO) 

.461 

-.012 

-.014 

.683 

Coding  Speed  (CS) 

.412 

.011 

.036 

.654 

Auto-Shop  Information  (AS) 

.402 

.776 

-.023 

.027 

-.089 

Math  Knowledge  (MK) 

.741 

-.069 

.003 

.037 

.480 

Mechanical  Comprehension  (MC) 

.632 

.488 

.017 

-.005 

.158 

Electronics  Information  (El) 

.547 

.569 

.061 

-.015 

Notes:. 

1.  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery, .  g  =  General  Intellectual  Ability. 

2.  Entries  in  bold  correspond  to  Promax  loadings  greater  than  .40 


Enhanced  Computer  Administered  Test  Factor  Structure 

Table  8  reports  the  results  of  the  hierarchical  solution  for  the  ECAT  battery.  Three  primary  factors 
were  found:  (1)  Spatial  Ability  defined  by  Integrating  Details,  Assembling  Objects,  Rgural  Reasoning,  and 
Spatial  Orientation;  (2)  Psychomotor  Skill  defined  by  the  One-  and  Two-Hand  Tracking  tests;  and 
(3)  Working  Memory  which  was  defined  by  Mental  Counters  and  Sequential  Memory.  This  factor  pattern 
roughly  matches  the  a  priori  categorization  of  the  tests  described  earlier,  except  that  the  Figural  Reasoning 
Test  loaded  higher  on  the  Space  factor  than  on  the  Memory  (i.e.  Nonverbal  Reasoning)  factor,  and  the 
fourth  construct  of  Perceptual  Speed  could  not  be  verified.  The  Target  Identification  test,  however,  did  not 
load  highly  on  any  of  the  first  three  factors.  If  the  ECAT  battery  had  included  additional  tests  of  perceptual 
speed,  it  is  likely  that  Target  Identification  would  have  loaded  on  a  fourth  Perceptual  Speed  factor.  All  of 
the  tests  loaded  on  the  general  factor  which  accounted  for  just  over  40%  of  the  correlational  variance. 
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Table  8 


Orthogonalized  Hierarchical  Solution  for  EC  AT 


Test 

Spatial 

Psychomotor 

Working 

Memory 

Mental  Counters  (CT) 

.690 

.130 

-.046 

.313 

Sequential  Memory  (SM) 

.643 

.019 

.000 

.583 

Figural  Reasoning  (FR) 

.703 

.210 

-.002 

.149 

Integrating  Details  (ID) 

.751 

.279 

-.018 

.009 

Assembling  Objects  (AO) 

.757 

.281 

-.036 

-.003 

Spatial  Orientation  (SO) 

.677 

.231 

-.057 

.033 

One-Hand  Tracking  (Tl) 

-.484 

.004 

.696 

-.017 

Two-Hand  Tracking  (T2) 

-.524 

-.017 

.716 

.009 

Target  Identification  (TI) 

-.402 

-.082 

.241 

-.021 

Notes. 

1.  ECAT  =  Enhanced  Computer  Administered  Testing,  g  =  General  Intellectual  Ability. 

2.  Entries  in  bold  correspond  to  Promax  loadings  greater  than  .40 


Enhanced  Computer  Administered  Test  and  Armed  Services  Vocational  Aptitude 

Battery  Factor  Structure 

Although  the  factor  patterns  appear  quite  different  across  the  batteries  this  does  not  directly 
address  the  question  of  the  degree  of  overlap  between  the  batteries.  A  partial  answer  to  this 
question  can  be  found  by  correlating  the  general  ability  scores  across  the  batteries.  The  range 
corrected  correlation  of  the  ECAT-g  and  ASVAB-g  scores  was  a  moderate  .71,  implying  that 
while  there  is  some  redundancy  across  the  batteries  there  is  substantial  uniqueness  as  well.  A  final 
factor  analysis  was  conducted  across  the  combined  test  batteries  (19  tests);  the  results  are 
reported  in  Table  9.  The  ASVAB  primary  factors  (Technical,  Verbal,  Clerical,  and  Math)  were 
relatively  unchanged  in  the  combined  analysis.  The  Psychomotor  factor  for  ECAT  also 
reemerged  intact  in  the  combined  analysis,  The  Space  factor  reemerged,  although  only  Integrating 
Details  and  Assembling  Objects  have  substantial  loadings  on  Space,  and  both  tests  have  higher 
loadings  on  another  factor.  The  ECAT  Working  Memory  factor  appears  to  have  captured  the 
nonverbal  reasoning  variance  in  many  of  the  ECAT  tests  and  thus  was  recast  as  a  nonverbal 
reasoning  factor.  This  latter  factor,  begging  the  point  that  it  is  at  the  primary  level,  appears  very 
much  like  a  fluid  intelligence  factor.  This  result  is  not  surprising.  Alderton  and  Larson  (1992a, 
1992b)  argue  that  the  net  effect  of  the  efforts  to  expand  the  ability  dimensions  measured  by  the 
ASVAB  was  to  augment  the  crystallized  intelligence  measures  of  the  ASVAB  with  fluid 
intelligence  measures,  thus  providing  a  more  complete  sampling  of  intellectual  performance.  Later 
in  this  report,  however,  we  will  present  evidence  showing  that  additional  specific  ability  factors 
are  also  required  to  maximize  incremental  validity. 
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Table  9 


Orthogonalized  Hierarchical  Solution  of  ASVAB  With  ECAT 


_ Test _ 

General  Science  (GS) 

Arithmetic  Reasoning  (AR) 

Word  Knowledge  (WK) 

Paragraph  Comprehension  (PC) 
Numerical  Operations  (NO) 
Coding  Speed  (CS) 

Auto-Shop  Information  (AS) 

Math  Knowledge  (MK) 
Mechanical  Comprehension  (MC) 
Electronics  Information  (El) 
Mental  Counters  (CT) 

Sequential  Memory  (SM) 

Figural  Reasoning  (FR) 
Integrating  Details  (ID) 
Assembling  Objects  (AO) 

Spatial  Orientation  (SO) 
One-Hand  Tracking  (Tl) 

Two- Hand  Tracking  (T2) 

Target  Identification  (TI) _ 


g 

Reas 

Tech 

Verb 

.740 

.012 

.161 

315c 

.762 

.212 

.152 

.070 

.706 

.016 

.024 

.605 

.637 

.028 

.025 

.449 

.406 

-.026 

.010 

-.019 

.371 

.033 

-.007 

.048 

.519 

.003 

.727 

-.023 

.699 

.139 

-.080 

.029 

.724 

.125 

.353 

.038 

.616 

-.024 

.477 

.091 

.571 

.537 

-.004 

-.041 

.536 

.534 

-.035 

.054 

.656 

.384 

-.001 

.076 

.661 

.371 

.049 

-.010 

.633 

.421 

.044 

-.023 

.630 

.316 

.099 

.023 

-.441 

-.013 

.017 

.014 

-.492 

-.014 

-.030 

-.004 

-.365 

-.115 

-.012 

-.048 

Motr 

Cler 

Math 

Spat 

-.024 

-.046 

.119 

.064 

.038 

.082 

358 

-.126 

-.004 

-.015 

-.049 

-.014 

.006 

.117 

.004 

-.042 

-.013 

.725 

.101 

-.018 

-.008 

.690 

-.047 

.077 

.011 

.018 

-.114 

-.040 

.009 

.096 

.425 

-.021 

-.058 

-.015 

.095 

.074 

-.001 

-.003 

.040 

.025 

-.037 

.051 

-.002 

-.036 

-.006 

.000 

-.028 

-.119 

-.012 

-.045 

.079 

.083 

-.024 

-.016 

.062 

.245 

-.008 

.049 

-.056 

388 

-.053 

-.050 

.056 

.134 

.736 

.003 

-.012 

.021 

.725 

.007 

.003 

-.001 

.232 

-.078 

.067 

-.152 

Notes. 

1.  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery,  ECAT  =  Enhanced  Computer  Administered  Testing,  g:=General  Intellectual 
Ability,  Motr:  =Psychomotor  Skill,  Reas:=  Nonverbal  Reasoning,  Clen  =Clerical  Speed,  Tech:  ^Technical  Knowledge,  Math: 
=Mathematical  Ability,  Verb:  =VerbaI  Ability,  Spat:  =Spatial  Ability. 

2.  Entries  in  bold  correspond  to  Promax  loadings  greater  than  .40 

The  fact  that  the  factor  analysis  of  the  combined  ASVAB  and  ECAT  battery  produced  three 
more  factors  than  ASVAB  alone  shows  that  the  ECAT  battery  indeed  does  measure  ability 
factors  not  adequately  measured  by  the  ASVAB.  Nevertheless,  the  overlap  between  the  batteries 
is  substantial,  as  shown  in  Table  10,  which  displays  the  correlations  between  the  ASVAB  and 
ECAT  factor  scores.  Even  within  the  ASVAB,  the  factor  scores  are  highly  correlated,  as 
illustrated  by  the  .72  correlation  between  Math  and  Verbal,  but  the  .71  correlation  between  Math 
and  Space  is  nearly  as  large. 
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Table  10 


Range-Corrected  Correlations  Among  ASVAB  and  ECAT  Factor  Scores 


Factor 

Verbal 

Math 

Technical 

Clerical 

Working 

Memory 

Space 

Psychomotor 

Verbal 

1.000 

.722 

.672 

.489 

.491 

.587 

-.365 

Math 

.722 

1.000 

.558 

.647 

.641 

.711 

-.405 

Technical 

.672 

.558 

1.000 

.166 

.387 

.603 

-.430 

Clerical 

.489 

.647 

.166 

1.000 

.472 

.420 

-.271 

Working  Memory 

.491 

.641 

.387 

.472 

1.000 

.789 

-.480 

Space 

.587 

.711 

.603 

.420 

.789 

1.000 

-.605 

Psychomotor 

-.365 

-.405 

-  .430 

-.271 

-.480 

-.605 

1.000 

Note.  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery.  ECAT  =  Enhanced  Computer  Administered  Testing. 


The  same  problem  is  evident  at  the  second-order  level  with  the  correlations  among  the  g 
factors  derived  from  the  different  batteries.  Table  1 1  shows  that  the  ECAT  and  ASVAB  g  factor 
scores  correlate  a  moderately  high  .71,  which  suggests  some  redundancy  across  the  batteries,  but 
which  implies  some  uniqueness  as  well. 


Table  11 

Range-Corrected  Correlations  Between  g  Factor  Scores 
From  Different  Batteries  (N  =  10,963) 


Battery 

ASVAB 

ECAT 

ASVAB  +  ECAT 

ASVAB 

1.000 

.707 

.948 

ECAT 

.707 

1.000 

.865 

ASVAB  +  ECAT 

.948 

.865 

1.000 

Note,  g  =  General  Intellectual  Ability,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery,  ECAT  =  Enhanced  Computer  Administered  Testing. 


Conclusions 

The  goal  in  selecting  tests  for  the  ECAT  battery  was  to  broaden  the  range  of  abilities 
measured  by  the  ASVAB,  the  rationale  being  that  this  would  maximize  the  probability  that  ECAT 
would  improve  the  ASVAB's  validity.  The  results  suggest  that  the  effort  was  largely  successful. 
While  the  two  g  measures  are  highly  correlated,  they  are  by  no  means  redundant.  Although  the 
ECAT  spatial  and  nonverbal  reasoning  factors  are  also  highly  correlated  with  several  ASVAB 
factors,  substantial  amounts  of  unique  variance  remain  which  may  improve  upon  the  ASVAB's 
validity.  Moreover,  the  ECAT  psychomotor  and  perceptual  speed  tests  are  nearly  independent  of 
the  ASVAB  and  thus  may  capture  aspects  of  training  and  job  performance  untouched  by  the 
ASVAB.  Finally,  the  ECAT  tests  have  less  adverse  impact  than  ASVAB  tests  because  they 
measure  fluid  intelligence,  rather  than  crystallized  knowledge  acquired  in  school. 
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Criteria  for  Validation 


Background 

In  1980,  the  Assistant  Secretary  of  Defense  (Manpower,  Reserve  Affairs  and  Logistics) 
directed  the  Military  Services  to  establish  a  research  and  development  program  to  link  enlistment 
standards  to  job  performance.  Some  of  these  job  performance  measurement  projects  were  still 
underway  at  the  beginning  of  the  ECAT  project,  and  we  were  fortunate  to  be  able  to  arrange  a 
cooperative  effort  with  the  Marine  Corps  to  administer  ECAT  tests  to  automotive  and  helicopter 
mechanics  at  the  same  time  that  job  knowledge  and  hands-on  job  performance  tests  were 
administered.  The  results  of  that  study  were  presented  by  Carey  (1994). 

The  samples  described  in  this  report  all  came  from  students  at  military  technical  training 
schools.  Instead  of  relying  on  Final  School  Grades,  as  has  been  traditional  for  most  validation 
studies  conducted  in  service  schools,  every  effort  was  made  to  collect  information  on  practical 
skills  taught  in  shop,  laboratory,  simulator,  or  other  exercises.  In  many  cases,  these  were  hands- 
on  performance  measures  similar  to  the  kinds  of  tests  used  in  Job  Performance  measurement 
projects. 

Kieckhaefer  et  al.  (1992)  describe  their  development  of  the  ECAT  criteria.  They  collected 
data  on  every  quiz,  homework  assignment,  and  laboratory/shop  exercise  for  samples  of  several 
hundred  students  at  each  school.  Based  on  factor  analysis,  they  constructed  composites  of  scores 
designed  to  measure  different  dimensions  of  achievement  in  each  school.  These  composites  will 
be  referred  to  as  “internal  school  criteria,”  because  they  are  seldom  published  outside  of  the 
school,  as  the  Final  School  Grades  are.  They  include  all  of  the  hands-on  performance  measures,  as 
well  as  composites  of  written  tests,  and  grades  on  each  learning  module.  Table  12,  Table  13,  and 
Table  14  list  the  77  criteria  that  were  used. 
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Table  12 


Criteria  in  Army  Courses  for  ECAT  Validation 

Location _ Code _ Title/Description _ 

Fort  Benning  11H  Heavy  Antiarmor  Weapons  Crewman 

11H(A)  HMMWV  Curriculum: 

TOALL  Slim  of  scores  on  8  Training  Objectives 

EVT1TO  TOW  Tracking  Time  on  Target  for  10  shots,  Event  1 

EVT2TO  TOW  Tracking  Time  on  Target  for  10  shots,  Event  2 

EVT3TO  TOW  Tracking  Time  on  Target  for  10  shots,  Event  3 

EVTSUM  Sum  of  Events  1-3  Scores 

TO_l  M966  TOW  Simulator  Tracking  Event  1  Total 

11H(B)  ITV  Curriculum: 

TOALL  Sum  of  scores  on  8  Training  Objectives 

EVT1TO  TOW  Tracking  Time  on  Target  for  10  shots.  Event  1 

EVT2TO  TOW  Tracking  Time  on  Target  for  10  shots.  Event  2 

EVT3TO  TOW  Tracking  Time  on  Target  for  10  shots,  Event  3 

E.VTSUM  Sum  of  Events  1-3  Scores 
TO_l  ITV  TOW  Simulator  Tracking  Event  1  Total 

TO_2  ITV  TOW  Simulator  Tracking  Event  2  Total 

TO_3  ITV  TOW  Simulator  Tracking  Event  3  Total 

ITVTOW  ITV  TOW  Simulator  Tracking  Total  Events  1-3 


Fort  Sill  13F  Field  Artillery  Fire  Support  Specialist 

MPRAD  Map  Reading  and  Radio  composite 

FIRING  Firing  composite 

FSG  Final  School  Grade 


Fort  Knox  19K  Tank  Crewman 

COMM  Communications  Performance 


WEAPON  Weapons  Maintenance  and  Preparation 

LANDNAV  Land  Navigation  and  Map  Reading 
LOADER  Load/Unload  main  tank  gun  and  machine  gun 
MAINT  Preventive  maintenance  and  trouble  shooting/repair 

NBC  Nuclear/Biological/Chemical  countermeasures 

AVERAGE  Mean  of  the  6  scores  above 


Note.  ECAT  =  Enhanced  Computer  Administered  Testing,  HMMWV  =  High  Mobility  Multipurpose  Wheeled 
Vehicle,  TOW  =  Tube-launched  Optically-tracked  Wire-guided  missile,  ITV  =  Improved  Tow  Vehicle, 
TO  =  Training  Objective. 
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Table  13 


Criteria  in  Navy  Schools  for  ECAT  Validation 


CDP 

School/Criteria 

Title/Description 

6278 

AC 

PERF 

Air  Traffic  Controller 

Mean  of  4  Performance  Tests 

6515 

AE 

SUM2 

Aviation  Electrician’s  Mate 

Average  of  Performance  Tests  loading  on  Factor  2 

6518 

AMS 

PERF 

Aviation  Structural  Mechanic  -  Structures 

Average  of  performance  tests  and  practical  work 

6506 

AO 

PRACTL 

Aviation  Ordnanceman 

Average  of  all  practical  work 

6239-41 

AV 

BSCAV 

ADVAV 

PERFORM 

Avionics  Technician 

Average  of  all  Basic  Avionics  Tests 

Average  of  all  Advanced  Avionics  Tests 

Average  of  all  Performance  Tests 

6070 

EM 

PHASE  1 

Electrician’s  Mate 

Average  of  all  Phase  I  tests 

6487 

EN 

Engineman 

603V 

ET(AEF) 

FSG 

FSG2 

PERF 

Electronics  Technician  -  Advanced  Electronics  Field 

Final  School  Grade  for  Phase  I 

Final  School  Grade  for  Phase  II 

Average  of  Phase  II  Performance  Tests 

609W 

FC 

RADAR 

Fire  Controlman 

Average  of  all  Radar  Tests 

6400 

GMG 

HALF1 

HALF2 

Gunner’s  Mate  -  Gun 

Average  of  Tests  1*14 
-  Average  of  Tests  14-27/30 

6492 

MM 

Machinist’s  Mate 

6540 

OS 

WRIT 

PERF 

Operations  Specialist 

Average  of  all  Written  Tests 

Average  of  all  Performance  Tests 

611E 

RM 

PHASE3 

Radioman 

Average  of  All  Knowledge  and  Performance  Tests  in  Last  Phase 

Notes.  1.  ECAT  =  Enhanced  Computer  Administered  Testing,  CDP  =  Course  Data  Processing  code. 
2.  FSG  (Final  School  Grade)  was  also  used  as  a  criterion  in  each  school. 
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Table  14 


Criteria  in  Air  Force  Schools  for  ECAT  Validation 


Location 

AFSC/Criterion 

Title 

Keesler  AFB 

73230 

Apprentice  Personnel  Specialist  (APS) 

ZHRS 

Standardized  training  hours  on  Blocks  II- VII 

AFPT70 

Air  Force  Performance  Test  Words  per  Minute  Typing 

FSG 

Final  School  Grade 

Keesler  AFB 

27230 

Apprentice  Air  Traffic  Control  Operator  (ATC) 

BLK2 

Control  Tower  Procedures  (Written  test  -  standardized  hours) 

BLK3A 

Basic  Control  Tower  Operation  (Perf  test  -  standardized  hours) 

BLK3B 

Advanced  Control  Tower  Operation  (Perf  test  -  standardized  hours) 

BLK5A 

Basic  Approach  Control  Operation  (Perf  test  -  standardized  hours) 

BLK5B 

Advanced  Approach  Control  Operation  (Perf  test  -  standardized  hours) 

FAA 

Federal  Aviation  Administration  Examination 

FSG 

Final  School  Grade 

Note.  ECAT  =  Enhanced  Computer  Administered  Testing,  AFSC  =  Air  Force  Specialty  Code. 


Several  difficulties  were  encountered  in  the  criterion  development. 

1.  Many  of  the  laboratory  exercises  were  scored  as  Pass/Fail,  with  almost  all  of  the  students 
passing.  Composites  derived  from  these  exercises  were  highly  skewed  with  small  variances. 

2.  Some  schools  had  alternative  tracks  or  major  curriculum  changes  during  the  course  of  the 
ECAT  study.  Criteria  that  were  available  for  one  curriculum  were  absent  in  the  next.  In  these 
cases,  the  sample  from  a  given  school  had  to  be  split  into  smaller  subsamples,  thus  reducing 
statistical  power. 

3.  The  Trank  Crewman  (19K)  school  was  selected  for  study  because  Smith  and  Graham  (1987) 
had  found  excellent  incremental  validities  with  a  combination  of  psychomotor  and  perceptual  tests 
for  predicting  performance  on  the  Unit  Conduct  of  Fire  Trainer  (UCOFT),  a  high-fidelity  tank 
gunnery  simulator.  Unfortunately,  budget  cuts  at  the  school  forced  this  simulator  to  be  shut  down. 
Other  available  criteria  turned  out  to  have  low  reliability. 

Because  of  minor  curriculum  changes  and  other  factors,  most  students  missed  one  or  more  of 
the  examinations  or  exercises  that  comprised  the  composite  criteria.  Therefore,  the  criteria  were 
defined  to  be  the  means  of  the  tests  or  exercises  actually  taken.  However,  these  means  were 
sometimes  bizarre  for  students  that  dropped  out  from  school  early  in  the  curriculum.  In  the  course 
of  the  data  analysis,  rules  were  formulated  to  reduce  the  number  of  outliers  due  to  missing  data. 
These  are  described  in  Appendix  E. 
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Criterion  Statistics 


Tables  G-l,  G-2,  and  G-3  show  the  basic  statistics  for  some  77  criteria  collected  at  three 
Army  schools,  two  Air  Force  Schools,  and  13  Navy  schools.  The  Army’s  1 1H  school  was  divided 
into  two  curricula,  labeled  11H(A)  and  11H(B).  The  Air  Force  Air  Traffic  Control  school  (ATC) 
had  two  curricula,  labeled  ATC(A)  and  ATC(B).  For  some  purposes,  these  were  combined  into  a 
single  group,  ATC.  In  the  tables,  the  digit  at  the  end  of  the  school  abbreviation  refers  to  the 
number  of  the  criterion.  Thus,  1 1H(A)  had  6  criteria,  while  1 1H(B)  had  9  criteria. 

Reliabilities  were  derived  from  those  computed  by  Kieckhaefer  et  al.  (1992)  In  several 
schools  (19K,  AV,  ET,  EN,  MM,  OS)  different  curricula  were  combined  into  one  group,  and  the 
weighted  average  of  the  reliabilities  of  corresponding  criteria  was  used. 

To  correct  the  criterion  means  and  standard  deviations  for  range-restriction,  Lawley’s  (1943) 
multivariate  range  correction  procedure  was  used,  with  all  10  ASVAB  tests  used  as  explicitly 
selected  variables.  The  corrected  reliability  was  computed  from  the  formula 

K»=0-7r)a-O. 

where  is  the  uncorrected  reliability,  is  the  uncorrected  standard  deviation,  and  the 
corresponding  corrected  values  are  in  upper  case  (Gulliksen,  1950/1987,  Chapter  10,  Eq.  5). 

Corrected  reliabilities  will  be  used  in  the  last  stage  of  correcting  validities.  Multiple 
correlations  are  first  tested  for  significance,  then  corrected  for  range  restriction,  then  corrected  for 
bias  using  the  Wheriy  formula,  and  finally  divided  by  the  square  root  of  the  corrected  reliability. 
In  cases  where  the  uncorrected  reliability  was  unknown  or  smaller  than  0.35,  no  correlation  for 
criterion  reliability  was  used.  Such  cases  are  designated  by  a  period  in  the  reliability  column  of  the 
tables. 

Selecting  Criteria  for  Meta-analysis 

For  some  purposes,  it  will  be  necessary  to  compute  mean  validities  or  combine  probabilities 
across  schools.  If  all  criteria  were  combined,  schools  with  the  most  criteria  would  receive  larger 
weights  in  the  averages.  Moreover,  the  criteria  within  a  school  are  not  independent,  thus 
complicating  the  analysis.  The  best  approach  seems  to  be  to  select  one  criterion  per  school  when 
combining  results  across  schools. 

Criterion  reliabilities  in  the  Army’s  19K  school  were  so  low  that  all  results  from  this  school 
were  reluctantly  dropped  from  the  meta-analysis. 

Four  sets  of  criteria  were  selected  for  averaging: 

1.  School  Grades  Because  school  grades  have  been  the  traditional  measures  of  training  success 
used  in  validation  studies,  we  felt  obligated  to  include  an  analysis  in  terms  of  school  grades,  even 
though  we  expected  some  of  the  ECAT  tests,  such  as  psychomotor  tracking,  to  have  no  relation 
to  the  kinds  of  written  tests  that  usually  form  the  basis  for  final  school  grades.  In  the  1 1H  Army 
school  where  FSG  was  unavailable,  a  summary  average  score,  EVTSUM,  was  used.  This  was 
actually  a  hands-on  performance  measure. 
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2.  Internal  School  Criteria  Among  the  measures  collected  for  each  school,  criteria  were  selected 
according  to  the  following  a  priori  rules,  which  were  applied  in  order: 

a. .  If  possible,  the  criterion  is  not  the  same  as  the  School  Grade.  If  there  is  no  other 
criterion,  then  School  Grade  is  chosen.  This  rule  tries  to  minimize  overlap  between  the 
Internal  Criteria  and  School  Grades. 

b. .  If  possible,  the  criterion  should  be  a  practical  performance  score,  rather  than  a 
knowledge  score. 

c. .  If  the  reliabilities  of  two  measures  are  substantially  different  (i.e.  by  0.10  or  more)  then 
the  more  reliable  one  is  used. 

d. .  A  measure  with  greatest  face  or  construct  validity  is  to  be  preferred  over  others. 

e. .  A  score  collected  late  in  training  should  be  preferred  to  one  taken  earlier. 

In  the  Army  1 1H  school,  after  Rule  (a)  was  applied,  rules  (d)  and  (e)  resulted  in  selecting 
the  TOW  simulator  tracking  performance  scores  taken  toward  the  end  of  training:  TO_l  and 
ITVTOW.  In  the  13F  school,  face  validity  favored  Firing  as  a  criterion.  In  the  Navy,  there 
was  never  more  than  one  performance  criterion  to  choose  from.  The  only  remaining  choice 
occurred  in  the  GM  school,  where  Rule  (e)  selected  a  measure  taken  in  the  last  half  of  the 
course  in  preference  to  the  earlier  measure.  In  the  Air  Force  APS  school,  the  only  practical 
performance  criterion  consisted  of  words  per  minute  on  the  AFPT70  typing  test. 

One  exception  to  the  a  priori  selection  was  made  in  the  Air  Force  Air  Traffic  Controllers, 
where  Rule  (c)  suggested  the  BLK3B  criterion  for  ATC(A)  and  the  BLK5A  criterion  for 
ATC(B).  However  in  order  to  maintain  consistency  with  ATC(B),  the  preferred  ATC(A) 
criterion  was  changed  to  BLK5A. 

3.  Final  School  Grades  for  10  Schools  with  Performance  Criteria  Ten  schools  had  both  FSG  and 
practical  performance  criteria  available.  For  those  samples,  the  corresponding  two  sets  of  criteria 
were  used  for  some  analyses.  The  10  schools  were  13F,  APS,  ATC,  AC,  AE,  AMS,  AO,  AV, 
ET,  and  OS. 

4.  Performance  Criteria  for  10  Schools  with  Final  School  Grades  These  are  practical  performance 
measures  on  the  same  10  schools  for  which  FSGs  were  available.  Because  the  Air  Force  ATC 
school  was  split  into  two  different  curricula,  the  number  of  samples  was  11. 

The  last  two  sets  of  criteria  are  subsets  of  the  first  two,  with  sample  sizes  only  a  third  of  the 
total.  However,  they  permit  a  strict  comparison  between  FSG  and  performance  criteria  for  the 
average  magnitude  of  incremental  validity3. 


’This  analysis  was  suggested  by  Dr.  Norm  Abrahams,  of  RGI,  Inc. 
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Validity  of  General  Ability  Factors 


Theoretical  Background 

It  has  been  argued  that  the  search  for  additional  tests  to  enhance  the  validity  of  ASVAB  is 
futile.  Many  previous  studies  have  shown  that  the  general  ability  factor,  or  “g”,  is  the  major 
predictor  of  school  and  job  performance,  and  the  ASVAB  is  an  excellent  measure  of  g  (Ree  & 
Earles,  1991;  Hunter,  1986). 

Others  have  argued  that  g  is  imperfectly  measured  by  the  ASVAB,  which  seems  to 
concentrate  on  verbal  and  numerical  crystallized  intelligence.  Enhancing  the  ASVAB  with  spatial 
and  reasoning  tests,  and  tests  which  require  fluid  intelligence,  should  produce  a  better  measure  of 
g  with  greater  predictive  validity. 

A  third  position  is  that  g,  although  important,  may  be  over-rated  for  predicting  certain  kinds 
of  jobs  requiring  special  abilities.  Hence,  additional  tests  of  spatial  and  psychomotor  ability,  for 
example,  should  have  incremental  validity  over  ASVAB  for  predicting  mechanical  repair, 
targeting,  vehicle  operations,  etc. 

Method 

Three  different  measures  of  g  were  developed,  using  the  matrix  of  range-corrected 
correlations. 

1.  Hierarchical  g  scores  were  computed  using  weights  derived  from  a  second-order  factor 
analysis  of  the  variables. 

2.  Principal  Component  g  was  computed  using  the  factor  score  weights  from  the  first 
principal  component.  This  method  should  maximize  the  variance  accounted  for  by  a  single 
factor. 

3.  Psychological  g  scores  were  computed  by  weighting  each  test’s  z-score  by  its  correlation 
with  the  Figural  Reasoning  test.  The  rationale  was  that  the  Figural  Reasoning  test  measures 
the  same  fluid  reasoning  ability  as  the  Raven  Progressive  Matrices  test,  which  often  has  been 
used  as  a  marker  for  g. 

Each  of  these  three  methods  were  applied  to  three  different  sets  of  variables:  ASVAB  only, 
ECAT  only,  and  ASVAB  +  EC  AT  combined. 
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The  validity  of  each  g  score  was  computed  for  each  criterion  and  corrected  for  multivariate 
range-restriction  and  criterion  unreliability.  The  validity  of  the  ASVAB  +  ECAT  g  was  compared 
with  the  validity  of  the  g  derived  from  ASVAB  alone.  The  significance  of  the  difference  was 
tested  with  an  asymptotic  test  for  range-corrected  dependent  multiple  correlations  (Hedges, 
Becker,  &  Wolfe,  1992,  Eq.  12).  When  applied  to  zero-order  correlations,  this  formula  is 
equivalent  to  comparing  the  value  of 


z  = 


(1  -  /f)2  +  c2  (1  -  r22  )2  -  c,c2[2r12  (1  -  rx  -  r22)  -  rxr2  (1  - 


with  the  (0,1)  normal  distribution,  where 

rx  is  the  uncorrected  validity  of  ASVAB  g, 

r2  is  the  uncorrected  validity  of  ASVAB  +  ECAT  g, 

r12  is  the  uncorrected  correlation  between  ASVAB  g  and  ASVAB  +  ECAT  g, 

Cj  is  the  ratio  of  range-corrected  to  uncorrected  validity  for  ASVAB  g, 

c2  is  the  ratio  of  range-corrected  to  uncorrected  validity  for  ASVAB  +  ECAT  g, 

and  n  is  the  sample  size. 

For  each  school,  a  probability  is  determined  by  comparing  the  value  of  z  with  the  upper  tail  of 
the  normal  distribution.  The  combined  probability  for  the  entire  sample  across  schools  is  given  by 

Schools 

the  chi-square  distribution  of  (-2  log  Pt )  with  2  x  Schools  degrees  of  freedom  (Fisher, 

i'=l 

1932).  The  degrees  of  freedom  are  36  for  the  School  Grade  set  and  38  for  the  Internal  School 
Criteria  Validities,  where  the  Air  Traffic  Control  school  was  split  into  two  groups. 

Mean  validities  were  weighted  averages  of  the  multiple  correlations  for  each  school, 
weighted  by  their  sample  sizes. 
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Results 


Table  15  shows  the  range-corrected  correlations  of  the  First  Principal  Component  scores 
with  Psychological  and  Hierarchical  g  factor  scores  for  the  three  data  sets.  It  is  apparent  the 
Psychological  g  is  virtually  identical  to  the  First  Principal  Component.  Hence  Psychological  g 
was  eliminated  from  the  subsequent  validity  analyses. 

Table  15 

Range-corrected  Correlations  of  g  With  First  Principal  Components 

(N  =  10,963) 


Battery 

Psychological  g 

Hierarchical  g 

ASVAB 

.999 

.966 

ECAT 

.994 

.985 

ASVAB  +  ECAT 

.998 

.987 

Note.  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery,  EC  AT  =  Enhanced  Computer  Administered  Testing. 

While  the  correlations  between  different  measures  of  g  within  the  same  battery  of  tests  are 
large,  the  correlations  between  batteries  are  considerably  lower,  as  was  shown  by  Table  11. 

Table  16  and  Table  17  present  the  validities  of  six  different  measures  of  g  for  School  Grades 
and  for  Internal  School  Criteria,  respectively.  All  validities  were  corrected  both  for  range- 
restriction  and  for  criterion  unreliability.  The  mean  validities  were  obtained  by  weighting  these 
values  by  the  sample  sizes. 

The  mean  validities  for  the  Hierarchical  g  were  smaller  than  the  First  Principal  Component 
validities  for  the  ASVAB  battery  only,  but  not  for  the  ECAT  only  or  ASVAB  +  ECAT  batteries. 
This  reversal  makes  interpretation  difficult  for  subsequent  comparisons.  For  the  First  Principal 
Component,  the  inclusion  of  ECAT  tests  does  not  significantly  improve  validity  for  predicting 
School  Grades,  while  for  the  Hierarchical  g,  ECAT  significantly  increases  validity  for  five  of  the 
school  samples  as  well  as  for  the  mean  across  schools  A  similar  result  was  obtained  for  the 
Internal  Criteria,  with  only  the  Hierarchical  g  showing  a  significant  increase  in  mean  validity. 

Discussion 

The  hypothesis  that  the  g  derived  from  a  broader  sampling  of  tests  contained  in  the 
ASVAB  +  ECAT  battery  will  have  greater  validity  seems  to  be  confirmed  by  these  data  for 
Hierarchical  g. 

As  we  shall  see,  full  least  squares  multiple  regression  produces  validities  that  average  .02  to 
.06  larger  than  those  for  g  for  all  batteries.  This  means  that  the  validity  partly  comes  from  specific 
ability  factors  relevant  to  specific  criterion  measures,  rather  than  from  the  general  ability  factor 
alone. 
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Table  16 


Corrected  Validities  of  Hierarchical  and  First  Principal  Component 
Measures  of  “G”  From  ASVAB,  ECAT,  and  ASVAB  +  EC  AT  Sets  of  Variables 

for  School  Grade  Criteria 


School 

Criterion 

N 

Index  of 

ASVAB 

ECAT 

ASVAB  +  ECAT 

Reliability 

Hier.  G 

P.C. 

Hier.  G 

P.C. 

Hier.  G 

P.C. 

11H(A)5 

EVTSUM 

546 

.981 

.330 

.354 

.368 

.376 

.387* 

.392 

11H(B)5 

EVTSUM 

316 

.981 

.358 

.388 

.394 

.425 

.416* 

.435 

13F1 

FSG 

821 

.894 

.744 

.774 

.719 

.709 

.791* 

.804 

APS1 

FSG 

446 

.933 

.817 

.812 

.674 

.649 

.812 

.797 

ATC1 

FSG 

484 

.914 

.713 

.722 

.621 

.624 

.733 

.729 

AC1 

FSG 

72 

.977 

.782 

.774 

.636 

.612 

.769 

.757 

AE1 

FSG 

278 

.961 

.639 

.669 

.616 

.607 

.687 

.691 

AMS1 

FSG 

244 

.965 

.799 

.843 

.641 

.631 

.820 

.803 

AOl 

FSG 

234 

.946 

.694 

.701 

.582 

.584 

.706 

.698 

AVI 

FSG 

544 

.983 

.762 

.772 

.668 

.654 

.789 

.776 

EMI 

FSG 

797 

.972 

.654 

.664 

.539 

.525 

.665 

.649 

EN1 

FSG 

750 

.956 

.707 

.751 

.590 

.582 

.743* 

.726 

ET2 

FSG2 

86 

.987 

.779 

.799 

.761 

.755 

.828 

.842 

FC1 

FSG 

778 

.983 

.793 

.817 

.655 

.632 

.813 

.791 

GM1 

FSG 

420 

.976 

.709 

.722 

.588 

.579 

.727 

.708 

MM1 

FSG 

801 

.948 

.505 

.546 

.464 

.452 

.546* 

.542 

OS1 

FSG 

713 

.955 

.768 

.768 

.665 

.649 

.772 

.771 

RM1 

FSG 

277 

.934 

.761 

.763 

.606 

.572 

.743 

.730 

Mean 

Grades 

8607 

.676 

.696 

.592 

.582 

.702** 

.695 

Notes.  1.  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery,  ECAT  =  Enhanced  Computer  Administered  Testing, 
Hier.  G  =  Hierarchical  g,  P.C.  =  first  principal  component. 

2.  For  definitions  of  schools  and  criteria,  see  Tables  12-14. 

*  p  <  .05  for  the  null  hypothesis  of  no  difference  from  the  corresponding  ASVAB  g.  **  p  <  .01 . 
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Table  17 


Corrected  Validities  of  Hierarchical  and  First  Principal  Component 
Measures  of  “G”  From  ASVAB,  EC  AT,  and  ASVAB  +  EC  AT  Sets  of  Variables 

for  Internal  School  Criteria 


Index  of 

ASVAB 

ECAT 

ASVAB  +  ECAT 

School 

Criterion 

N 

Reliability 

Hier.  G 

P.C. 

Hier.  G 

P.C. 

Hier.  G 

P.C. 

11H(A)6 

TO_l 

542 

.945 

.216 

.221 

.212 

.234 

.239 

.244 

11H(B)9 

ITVTOW 

318 

.960 

-.018 

-.008 

.078 

.095 

.046 

.041 

13F3 

FIRING 

821 

.814 

.690 

.711 

.612 

.611 

.713 

.718 

APS3 

AFPT70 

432 

.965 

.318 

.286 

.315 

.278 

.291 

.308 

ATC(A)4 

BLK5A 

205 

.711 

.432 

.482 

.624 

.633 

.549 

.598 

ATC(B)4 

BLK5A 

295 

.911 

.390 

.375 

.518 

.532 

.455 

.485 

AC2 

PERF 

76 

.825 

.287 

.269 

.497 

.476 

.345 

.398 

AE2 

SUM2 

273 

.916 

.548 

.587 

.545 

.532 

.601 

.607 

AMS2 

PERF 

244 

.880 

.532 

.598 

.553 

.546 

.602 

.620 

A02 

PRACTL 

229 

.871 

.453 

.448 

.435 

.437 

.458 

.478 

AV4 

PERFORM 

352 

.820 

.507 

.561 

.561 

.558 

.608 

.605 

EM2 

PHASE1 

797 

.943 

.687 

.687 

.  .570 

.552 

.697 

.676 

EN1 

FSG 

750 

.956 

.707 

.751 

.590 

.582 

.743* 

.726 

ET3 

PERF 

86 

.941 

.643 

.635 

.632 

.665 

.684 

.700 

FC2 

RADAR 

780 

.891 

.682 

.731 

.543 

.540 

.712 

.692 

GM3 

HALF2 

397 

.959 

.687 

.721 

.561 

.551 

.709 

.693 

MM1 

FSG 

801 

.948 

.505 

.546 

.464 

.452 

.546* 

.542 

OS3 

PERF 

815 

.896 

.721 

.741 

.714 

.705 

.766* 

.784* 

RM2 

PHASE3 

277 

.832 

.679 

.689 

.575 

.547 

.685 

.674 

Mean 

Internal 

8490 

.556 

.579 

.517 

.511 

.591** 

.592 

Notes.  1.  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery,  ECAT  =  Enhanced  Computer  Administered  Testing, 
Hier.  G  =  Hierarchical  g,  P.C.  =  first  principal  component. 

2.  For  definitions  of  schools  and  criteria,  see  Tables  12-14. 

*  p  <  .05  for  the  null  hypothesis  of  no  difference  from  the  corresponding  ASVAB  g.  **  p  <  .01 . 
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Full-Model  Regression  Analysis  for  Each  Criterion 


Rationale 

This  section  will  compare  the  validities  of  regression  equations  based  on  the 
ASVAB  +  EC  AT  battery  with  equations  based  on  ASVAB  alone  to  determine  how  much  the 
addition  of  ECAT  tests  can  improve  validity.  The  use  of  optimally-weighted  ASVAB  tests  in 
regression  is  controversial.  Integer-weighted  ASVAB  composites  have  been  used  operationally  in 
preference  to  regression  equations.  In  part  this  is  an  historical  accident,  dating  from  the  times  of 
hand  calculation.  However,  regression  equations  may  involve  negative  weights,  penalizing  high 
test  scores,  and  may  not  cross-validate  as  well  as  simpler  weighting  schemes  on  small  samples. 
Some  people  believe  that  ECAT  tests  should  be  evaluated  against  the  operational  composites  by 
comparing  the  validity  of  the  operational  composite  with  an  optimally  weighted  combination  of 
the  ECAT  tests  and  the  operational  composite.  Unfortunately,  this  method  would  quickly  show 
that  the  ASVAB  itself  has  incremental  validity  over  the  operational  composite.  In  other  words,  an 
“ECAT’  test  that  was  merely  an  alternate  form  of  an  existing  ASVAB  test  would  appear  to  have 
incremental  validity,  because  the  operational  composite  is  suboptimal,  and  does  not  represent  the 
full  predictive  power  of  the  ASVAB.  We  must  reject  the  fallacy  of  such  an  approach. 

Others  have  suggested  using  stepwise  regression,  so  that  only  the  significant  ASVAB 
predictors  for  a  particular  criterion  are  used  to  establish  the  restricted  model.  The  problem  with 
this  approach  is  more  subtle:  the  degrees  of  freedom  for  significance  testing  will  be  incorrect. 
Stepwise  regression  capitalizes  on  chance  by  selecting  the  best  predictors  from  a  larger  pool. 
Significance  tests  will  be  biased  unless  the  number  of  predictors  in  the  larger  pool  are  used  for  the 
degrees  of  freedom.  But  in  that  case,  one  might  as  well  use  all  of  the  predictors  to  begin  with. 

It  is  important  to  realize  that  we  are  recommending  and  using  regression  for  analysis,  not 
prediction.  The  best  equation  for  prediction  is  not  the  sample  regression  equation,  but  some 
variation  on  it,  perhaps  ridge  regression,  perhaps  integer  weighted  composites.  Since  the 
appropriate  prediction  equation  has  not  yet  been  determined,  cross-validation  is  premature,  and  is 
not  covered  here.  Instead,  regression  will  be  used  to  test  hypotheses  and  estimate  population 
validities  of  population  regression  equations  from  their  sample  values. 

Method 


Significance  Testing 

For  each  criterion  in  every  school,  the  multiple  correlation  was  computed  using  the  10 
ASVAB  tests  as  predictors.  A  second  multiple  correlation  was  determined  by  all  10  ASVAB  tests 
plus  6  additional  ECAT  predictors:  Memory  Composite  (the  sum  of  z-scores  for  Mental  Counters 
and  Sequential  Memory),  Spatial  Composite  (the  sum  of  z-scores  of  Integrating  Details  and 
Assembling  Objects),  Tracking  Composite  (the  sum  of  z-scores  for  One-Hand  and  Two-Hand 
Tracking),  Figural  Reasoning,  Target  Identification,  and  Spatial  Orientation.  The  tests  in  the 
composites  were  chosen  because  a  priori  inspection  of  the  contents  of  the  tests  suggested  that 
each  pair  in  a  composite  was  measuring  the  same  construct.  Moreover,  each  test  in  a  pair 
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correlates  higher  with  the  other  test  than  it  does  with  any  other  test,  as  shown  in  Tables  B- 1  and 
B-2  of  Appendix  B.  For  each  criterion,  the  probability  associated  with  the  difference  was 
determined  from  the  F-distribution  with  degrees  of  freedom  equal  to  6  and  N  -  (10  +  6)  -  1, 
where 


A R2 


N  —  17 


6,n-\i 


1  -R' 


ASVAB+ECAT 


The  composites  were  used  instead  of  the  nine  ECAT  individual  tests  in  order  to  decrease  the 
degrees  of  freedom  in  the  numerator  and  thus  increase  the  statistical  power  of  the  F-test. 

Correction  for  Range  Restriction 

Next,  the  correlation  matrix  of  predictors  and  criterion  was  corrected  for  multivariate  range- 
restriction,  using  a  two  stage  process.  First,  the  uncorrected  matrix  of  10  ASVAB  tests  and  9 
ECAT  tests  was  obtained  for  the  sample  of  10,963  subjects  in  the  ECAT  sample.  This  matrix  was 
corrected  for  range  restriction  using  the  10  ASVAB  variables  as  explicitly  selected  variables  with 
population  covariances  equal  to  those  of  the  1991  joint- services  applicant  population  (N  = 
650,278).  The  population  matrix  is  shown  in  Table  A-l  of  Appendix  A. 

The  corrected  19x19  matrix  (Table  4)  was  treated  as  if  it  were  the  population.  The 
correlation  matrix  of  ASVAB  and  ECAT  tests  and  a  school  criterion  was  then  corrected  for  range 
restriction  as  if  all  19  tests  were  explicitly  selected.  This  method  has  the  advantage  of  having  a 
common,  shared  matrix  of  predictor  covariances  among  schools. 

After  correcting  the  correlation  matrix,  the  multiple  correlations  were  recomputed  for  the  10- 
predictor  and  16-predictor  models. 

Estimating  the  Population  Multiple  R2 

The  sample  multiple  correlations  were  “shrunken”  to  produce  unbiased  estimates  of  the 
population  R2  using  the  Wherry  formula.  However,  negative  estimates  of  R2  were  replaced  by 
zero,  which  re-introduced  some  bias.  Finally,  shrunken  multiple  correlations  were  generated  by 
taking  the  square  roots  of  the  shrunken  R2. 

Correction  for  Criterion  Unreliability 

The  shrunken  range-corrected  correlations  were  divided  by  the  square  roots  of  the  range- 
corrected  criterion  reliabilities  to  produce  the  “fully  corrected”  multiple  correlations.  The 
incremental  validities  were  the  differences  between  fully  corrected  multiple  correlations  for  the 
ASVAB-only  and  ASVAB  +  ECAT  regression  models.  The  “percent  increase”  was  defined  as 
100  times  the  validity  increment  divided  by  the  fully-corrected  ASVAB  multiple  correlation. 
Where  the  fully-corrected  ASVAB  validity  was  zero,  the  percent  increase  was  undefined. 
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Combining  Results  Across  Samples 

Using  one  criterion  per  school,  the  results  were  combined  across  samples  for  the  School 
Grade  criterion  set.  Another  analysis  was  done  for  the  set  of  Internal  School  Criteria. 

Schools 

The  combined  probability  is  given  by  the  chi-square  distribution  of  ^(—2 log/*)  with 

i=i 

2  x  Schools  degrees  of  freedom  (Fisher,  1932).  The  degrees  of  freedom  are  36  for  the  School 
Grade  set  and  38  for  the  Internal  School  Criteria  Validities,  where  the  Air  Traffic  Control  school 
was  split  into  two  groups. 

Mean  validities  were  weighted  averages  of  the  multiple  correlations  for  each  school, 
weighted  by  their  degrees  of  freedom,  N  -  p  -  1,  where  p  =  the  number  of  predictors  in  the 
regression  equation 

Weighting  by  degrees  of  freedom  is  contrary  to  the  method  advocated  by  Hedges,  Becker 
and  Wolfe  (1992),  who  recommend  weighting  each  correlation  by  the  inverse  of  its  asymptotic 
variance.  However,  two  anomalies  were  observed  when  using  the  variances  for  weighting  the 
multiple  R's. 

1.  For  a  subsample  of  ECAT,  the  mean  of  the  uncorrected  full  model  multiple  R's  was  .956, 
even  though  17  of  the  19  schools  had  multiple  R's  below  .67.  It  turned  out  that  performance 
criteria  for  two  schools,  AC  and  ET,  had  only  19  cases  with  16  predictors  in  the  full  model, 
leaving  multiple  R's  of  .995  and  .983  respectively.  These  inflated  multiple  R’s  produced 
variance  estimates  of  .000007  and  .000076  respectively.  The  low  variances  resulted  in  huge 
weights  for  these  schools  when  averaging  took  place.  Although  one  can  argue  that  the 
correct  "fix"  would  be  to  throw  out  schools  with  such  small  samples,  this  experience  suggests 
that  variance  weighting  may  distort  the  averages  obtained  when  even  one  multiple  R  is 
inflated. 

2.  Hedges,  Becker,  and  Wolfe  recommend  that  the  variance  of  the  range-corrected  multiple 
Rc  be  estimated  by  multiplying  the  variance  of  the  uncorrected  Ru  by  the  factor  c2,  where 

R 

c  —  — — .  For  the  AO  school  practical  criterion  ( N  =  132),  the  simple  validity  of  Two-Hand 

K 

Tracking  was  -.00896,  corrected  to  .00027.  The  obtained  value  of  c  was  0.0300,  resulting  in 
an  estimate  of  the  variance  of  the  corrected  R  of  .000009  and  a  weight  of  111,790  for 
averaging  purposes.  The  resulting  mean  validity  of  Two-Hand  Tracking  was  -.004,  even 
though  several  large  schools  had  validities  greater  than  .30  in  absolute  value. 

Based  on  these  experiences,  weighting  multiple  correlations  by  their  degrees  of  freedom 
appears  to  be  the  safest  method  of  averaging  multiple  correlations. 
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Results 


Incremental  Validities  of  ECAT  Tests 

Incremental  validities  for  all  Army,  Air  Force,  and  Navy  criteria  are  presented  in  Tables  1-1, 
1-2,  and  1-3  of  Appendix  I.  The  incremental  validities  for  School  Grade  criteria  appear  below  in 
Table  18.  Table  19  presents  the  results  for  the  Internal  School  Criteria. 

The  combined  probability  values  are  less  than  1.4  xlO-17  for  all  sets  of  criteria,  indicating  the 
overall  findings  are  highly  significant.  The  mean  corrected  validity  increase  was  .015  for  School 
Grades  and  .031  for  the  Internal  criteria.  However,  the  Grades  criteria  actually  included  the 
EVTSUM  performance  criteria  for  the  Army’s  11H  school,  which  showed  very  large  validity 
increments.  Seven  of  the  13  Navy  schools  showed  no  significant  validity  increment  in  Grades,  and 
nine  Navy  schools  showed  no  significant  increment  for  the  Internal  School  criteria.  Where  the 
criteria  exhibited  significant  validity  increments,  they  were  often  quite  large,  but  their  effects  were 
diluted  when  averaged  with  zero-effect  criteria. 

Table  20  compares  the  summary  results  for  pure  FSG  criteria  with  those  for  pure  practical 
performance  measures.  The  mean  validity  gain  for  predicting  FSG  is  1.7%,  while  the  gain  for 
predicting  performance  measures  is  5.7%.  These  findings  are  consistent  with  those  reported  by 
Wolfe  and  Alderton  (1992)  and  Wolfe,  Alderton,  and  Larson  (1993)  for  a  related  battery 
administered  to  recruits  for  nine  Navy  schools.  The  mean  incremental  validity  for  predicting  FSG 
is  about  the  same  as  those  reported  for  Project  A  Core  Technical  Proficiency  by  McHenry, 
Hough,  Toquam,  Hanson,  and  Ashworth  (1990),  but  the  validity  reported  for  the  ASVAB  was 
only  .63  in  their  study. 

The  mean  validity  increments  are  quite  substantial,  particularly  for  the  performance  criteria. 
If  these  means  were  representative  of  all  military  training  schools,  they  would  provide  strong 
evidence  for  the  utility  of  enhancing  the  ASVAB  with  new  tests. 

For  both  FSG  and  Internal  Criteria,  all  but  one  of  the  Air  Force  and  Army  schools  had  above- 
average  validity  increments,  while  10-11  out  of  13  Navy  schools  were  below  average.  The  best 
Army  results  were  in  the  11H  Heavy  Antiarmor  Weapons  simulator  training  performance,  where 
the  validity  increase  was  0.24  correlation  points.  In  the  Air  Force,  both  Air  Traffic  Controller  and 
Personnel  Specialist  showed  validity  gains,  but  the  increments  were  about  four  times  greater  for 
performance  criteria  (as  large  as  0.10  for  ATC  Basic  Approach  Control  Operations).  In  the  Navy, 
the  largest  significant  validity  improvement  was  0.031  for  Operations  Specialist  performance. 
The  large  nonsignificant  .149  validity  increase  for  Navy  Air  Traffic  Control  performance  should 
be  noted,  since  it  is  consistent  with  the  large  and  significant  improvements  in  predicting  Air  Force 
Air  Traffic  Control  Performance. 

Incremental  Validities  of  Ability  Factors 

In  order  to  guard  against  overestimating  incremental  validity  by  underestimating  the 
predictive  validity  of  the  ASVAB,  it  has  been  necessary  to  represent  the  ASVAB  by  using  all  10 
of  its  subtests  as  predictors  in  the  regression  equations.  Nevertheless,  many  of  the  results  are 
more  interpretable  in  terms  of  ability  factors,  rather  than  individual  tests.  Therefore,  the  entire 
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analysis  was  repeated,  using  a  4-factor  score  representation  of  the  ASVAB  and  a  3-factor 
representation  of  the  ECAT  battery.  Each  criterion  was  first  fit  to  a  regression  equation  with  the 
four  ASVAB  factor  scores  as  independent  variables.  Then  three  ECAT  factor  scores  were  added, 
making  a  7-predictor  regression  equation.  The  results  for  each  school  were  remarkably  similar  to 
those  reported  in  Table  18,  Table  19,  and  Table  20.  The  mean  incremental  validities  are  shown  in 
Table  21,  and  should  be  compared  with  the  bottom  lines  of  Table  18,  Table  19,  and  Table  20 
(See  Appendix  J  for  further  details  on  factor  validities.)  Although  the  validities  are  slightly  less  for 
factor  scores,  the  incremental  validities  are  about  the  same,  except  for  Performance  criteria,  which 
show  larger  gains  with  the  factor  score  representation. 

Figure  10  conveniently  summarizes  the  incremental  validities  obtained  from  different  forms  of 
predictor  representation.  It  is  immediately  clear  that  the  incremental  validity  is  about  the  same, 
whether  in  terms  of  tests,  factors,  or  Hierarchical  g.  This  suggests  that  the  validity  improvement 
from  ECAT  is  due  to  a  better  sampling  of  the  tests  comprising  g.  However,  the  same  figure  also 
shows  that  g  misses  more  predictive  validity  than  ECAT  adds,  that  is,  the  validity  of  enhanced  g 
is  less  than  the  validity  of  the  four  ASVAB  factors  combined  into  a  multiple  regression  equation. 
Thus,  using  g  for  prediction  wastes  a  significant  part  of  the  ASVAB’s  validity,  contrary  to  the 
point  of  view  expressed  by  Ree  and  Earles  (1991).  On  the  other  hand,  basing  prediction  on  factor 
scores  is  nearly  as  effective  as  using  individual  tests,  and  may  be  an  excellent  way  of  reducing  the 
errors  associated  with  using  too  many  predictors  in  regression,  while  retaining  the  potential  for 
differential  prediction. 
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Table  18 


ECAT  Incremental  Validities  for  School  Grades 


Uncorrected  Multiple  R _  Corrected  Multiple  R 


School 

Criterion 

Sample 

Size 

ASVAB 

ASVAB  Percent 
+ECAT  Variance 

Probability 

of/W 

ASVAB 

Increase 

Percent 

Increase 

11H(A)5 

EVTSUM 

546 

.321 

.373 

4.119 

1.53xl0'3 

.392 

.036 

9.2  ** 

11H(B)5 

EVTSUM 

316 

.330 

.446 

11.216 

1.64xl0"5 

.382 

.091 

23.7  ** 

13F1 

FSG 

821 

.544 

.597 

9.483 

9.81  xlO"14 

.790 

.024 

3.0** 

APS1 

FSG 

446 

.545 

.581 

6.233 

2.17xl0'4 

.828 

.012 

1.5  ** 

ATC1 

FSG 

484 

.403 

.445 

4.540 

1.98xl0"3 

.727 

.020 

2.7  ** 

AC1 

FSG 

72 

.627 

.649 

4.978 

8.37x10"' 

.839 

.000 

0.0 

AE1 

FSG 

278 

.489 

.542 

7.810 

3.04xl0"3 

.659 

.023 

3.5  ** 

AMS1 

FSG 

244 

.599 

.602 

.555 

9.73x10"' 

.848 

.000 

0.0 

AOl 

FSG 

234 

.504 

.522 

2.434 

5.10x10“' 

.717 

.005 

0.7 

AVI 

FSG 

544 

.517 

.536 

2.772 

2.49X10"2 

.810 

.005 

0.7* 

EMI 

FSG 

797 

.451 

.459 

.864 

3.47x10"' 

.687 

.000 

0.0 

EN  1 

FSG 

750 

.584 

.588 

.721 

5.09x10"' 

.763 

.000 

0.0 

ET2 

FSG2 

86 

.504 

.566 

9.738 

3.60x10"' 

.813 

.027 

3.3 

FC1 

FSG 

778 

.499 

.528 

4.180 

2.28  xlO"5 

.828 

.010 

1.2** 

GM1 

FSG 

420 

.428 

.454 

2.911 

7.10xl0“2 

.731 

.004 

0.6 

MM1 

FSG 

801 

.402 

.425 

2.362 

5.41  xlO"3 

.557 

.012 

2.2  ** 

OS1 

FSG 

713 

.565 

.582 

2.969 

2.33  xlO"3 

.804 

.007 

0  9** 

RM1 

FSG 

277 

.536 

.587 

8.796 

1.17X10"3 

.775 

.022 

2.8** 

Summary 

Grades 

8607 

.467* 

.510 

4.194b 

<1.4  xlO-17' 

.713 

.015 

2.0d** 

Notes.  1.  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery, 
FSG  =  Final  School  Grade. 

2.  For  definitions  of  schools  and  criteria,  see  Tables  12-14. 
aMean  multiple  Rs  are  means  of  Wherry -shrunken  Rs. 

.  A R2 

Percent  Variance  =  lOOx - j - 

1  —  R  ASVAB+ECAT 

cSummary  probability  = 

dThe  summary  percent  increase  is  defined  as  100  X  the  ratio  of  the  mean  increase  to  the  mean  corrected  ASVAB  validity. 
*  p  <  .05  for  uncorrected  R  increase.  **  p  <  .01  for  uncorrected  R  increase. 
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Table  19 


ECAT  Incremental  Validities  for  Internal  School  Criteria 


Uncorrected  Multiple  R 

Corrected  Multiple  R 

Sample 

ASVAB 

Percent 

Probability 

Percent 

School 

Criterion 

Size 

ASVAB 

+ECAT  Variance 

of  Fw  -i7 

ASVAB 

Increase 

Increase 

11H(A)6 

TO_l 

542 

.210 

.269 

3.031 

1.52X10"2 

.240 

.046 

19.1  * 

11H(B)9 

ITVTOW 

318 

.154 

.350 

11.203 

1.51xl0"5 

.075 

.237 

316.3  ** 

13F3 

FIRIA  G 

821 

.444 

.466 

2.507 

2.82X10'3 

.730 

.007 

1.0  ** 

APS3 

AFPT70 

432 

.294 

.404 

9.129 

2.28X10-6 

.388 

.079 

20.4  ** 

ATC(A)4 

BLK5A 

205 

.322 

.404 

7.127 

4.18X10'2 

.614 

.079 

12.9* 

ATC(B)4 

BLK5A 

295 

.312 

.408 

8.316 

1.04X10"3 

.450 

.100 

22.2  ** 

AC2 

PERF 

76 

.330 

.460 

13.033 

2.80  xlO'1 

.381 

.149 

39.2 

AE2 

SUM2 

273 

.440 

.487 

5.808 

2.39xl0“2 

.608 

.022 

3.7* 

AMS  2 

PERF 

244 

.393 

.431 

3.892 

1.89X10"1 

.650 

.016 

2.4 

A02 

PRACTL 

229 

.343 

.374 

2.652 

4.69X10'1 

.490 

.010 

2.1 

AV4 

PERFORM 

352 

.379 

.409 

2.853 

1.48X10"1 

.673 

.016 

2.4 

EM2 

PHASE1 

797 

.474 

.482 

.950 

2.86X10-1 

.729 

.001 

0.1 

EN  1 

FSG 

750 

.584 

.588 

.721 

5.09X10'1 

.763 

.000 

0.0 

ET3 

PERF 

86 

.482 

.574 

14.533 

1.41  xlO”1 

.735 

.075 

10.2 

FC2 

RADAR 

780 

.345 

.381 

3.053 

7.93x10^ 

.733 

.016 

2.1  ** 

GM3 

HALF2 

397 

.458 

.467 

1.033 

6.87X10-1 

.734 

.000 

0.0 

MM1 

FSG 

801 

.402 

.425 

2.362 

5.41  xlO"3 

.557 

.012 

2.2** 

OS3 

PERF 

815 

.523 

.564 

6.510 

3.81X10"9 

.791 

.025 

3.1  ** 

RM2 

PHASE3 

277 

.420 

.464 

4.907 

5.08X10"2 

.702 

.017 

2.4 

Summary 

Internal 

8490 

.373a 

.440 

3.966b 

<  1.4x10"17c 

.619 

.031 

5.0d  ** 

Notes.  1.  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery, 
FSG  =  Final  School  Grade. 

2.  For  definitions  of  schools  and  criteria,  see  Tables  12-14. 
aMean  multiple  Rs  are  means  of  Wherry-shrunken  Rs. 

.  A R2 

Percent  Variance  =  lOOx - - - 

1““^?  ASVAB+ECAT 

cSummary  probability  =  ^(X^)* 

dThe  summary  percent  increase  is  defined  as  100  x  the  ratio  of  the  mean  increase  to  the  mean  corrected  ASVAB  validity. 
*  p  <  .05  for  uncorrected  R  increase.  **  p  <  .01  for  uncorrected  R  increase. 
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Table  20 


Comparison  of  FSG  and  Performance  Criteria  for  10  Schools  With  Both  Criteria 


Uncorrected  Multiple  R _  Corrected  Multiple  R 


Criterion 

Set 

Sample 

Size 

ASVAB3 

ASVAB 

+ECAT 

Percent 

Variance15 

Probability0 

of  F6JV  -17 

ASVAB 

ECAT 

Increase 

Percent 

Increase 

FSG 

3922 

0.505 

0.551 

5.219 

4.163X10"17 

0.013 

1.7* 

Performance 

3828 

0.373 

0.453 

2.442 xlO-15 

0.036 

5.7* 

Notes.  FSG  =  Final  School  Grade,  ASVAB  =  Aimed  Services  Vocational  Aptitude  Battery,  ECAT  =  Enhanced  Computer 
Administered  Testing. 

aMean  multiple  Rs  are  means  of  Wherry-shrunken  Rs. 

.  A R2 

bPercent  Variance  =  lOOx - - - 

1  —  7?  ASVAB  +  ECAT 

cSummary  probability  =  P(X^)  for  FSG  and  /^(X^  )  for  Performance. 

*  p  <  10'14  for  uncoirected  R  increase. 


Table  21 

Incremental  Validities  of  3  ECAT  Factor  Scores  Over  4  ASVAB  Factor  Scores 

Uncorrected  Multiple  R  Corrected  Multiple  R 

Criterion  Sample  ASVAB  Percent  ECAT  Percent 


Set 

Size 

ASVAB3 

+ECAT 

Variance15 

ASVAB 

Increase 

Increase0 

Internal 

8490 

.342 

.413 

3.370 

.608 

.031 

5.1  * 

Grades 

8607 

.447 

.488 

3.325 

.708 

.013 

1.9* 

Performance 

3828 

.328 

.424 

5.374 

.620 

.041 

6.6* 

FSG 

3922 

.487 

.534 

4.838 

.778 

.014 

1.9* 

Notes.  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude 
Battery,  FSG  =  Final  School  Grade. 

‘Mean  multiple  Rs  are  means  of  Wherry-shrunken  Rs. 

.  AT?2 

Percent  Variance  =  Mean  of  100  x - — - - 

1  —  7?  ASVAB  +  ECAT 

cSummary  probability  =  PQi^)  for  Grades,  /^(X^g)  for  Internal  Criteria,  ^(X^)  for  FSG,  and  P(X^)  for 

Schools 

Performance,  where  Schools  X-21og(P(F3>JVi_8)). 

k-\ 


*  p  < 


1.4xl0"17 


for  uncorrected  R  increase. 


44 


ECAT  Mecn  Vdiclfy  Increase  for  Irtemd  Criteria 


0.66 

0.64 


Her.  G  IstP.C  Factors  Tests 

Predcta  Repesertdicn 

Figure  10.  ASVAB  validities  and  ECAT  validity  increases  for  different  predictor  representations. 
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Regression  Analyses  for  Each  Predictor’s  Incremental  Validity 


Method 


Significance 

For  each  criterion  in  each  school  and  each  predictor,  two  hypotheses  were  tested: 

1.  Adding  just  this  one  predictor  to  the  ASVAB  does  not  significantly  increase  the  multiple  R: 


A R  =  R2 ASVAB+l  —  R2 ASVAB  . 


^l.AT-12  =  (N  —  12) 


AR 


1  -R' 


ASVAB+ 1 


2.  Deleting  just  this  one  predictor  from  the  full  battery  of  ASVAB  plus  ECAT  does  not 
significantly  decrease  the  multiple  R: 

The  general  approach  was  to  examine  the  validity  of  composites  first;  if  significant,  then  the 
component  tests  in  a  composite  were  examined  for  significance.  When  it  was  necessary  to  split  a 
composite  up  into  its  components,  then  the  remaining  composites  were  also  split.  Thus,  if  the 
predictor  was  a  test  that  was  also  part  of  a  composite,  then  the  battery  of  ASVAB  plus  nine 
individual  tests  was  compared  with  the  same  battery  with  the  one  test  deleted;  otherwise,  the  full 
battery  consisting  of  ASVAB  plus  three  composites  plus  three  tests  was  compared  with  the  same 
battery  with  the  one  predictor  deleted.  For  example,  the  effect  of  deleting  One-Hand  tracking 
was  tested  by  comparing  the  multiple  R  of  ASVAB  plus  nine  ECAT  predictors  with  the  Multiple 
R  of  ASVAB  plus  eight  ECAT  predictors. 

AR2=R 2  ' ASVAB+9ECAT  —  R2 ASVAB+ZECAT  . 


^.tf-20  =  (N-20) 


AR2 


1  -R' 


ASVAB +9  ECAT 


The  effect  of  deleting  Target  Identification  was  tested  by  comparing  the  Multiple  R  of 
ASVAB  plus  six  ECAT  predictors  with  the  Multiple  R  of  ASVAB  plus  five  ECAT  predictors. 


AR2  =  R 7  '  ASVAB+6ECAT  —  R2  ASVAB +5 ECAT  . 


B'l.N-n  =  (N_17) 


AR2 


1  -R' 


ASVAB+6ECAT 
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Corrected  Multiple  Correlations  and  Increments 

After  significance  testing,  all  correlations  were  corrected  for  multivariate  range  restriction, 
shrunken  to  their  population  values  (Wherry),  and  corrected  for  criterion  unreliability. 


Combining  Results  Across  Samples 

For  the  two  subset  of  criteria  (School  Grades  and  Internal  School  Criteria),  the  Fisher  chi- 
square  method  was  used  to  combine  probabilities.  Mean  validities  were  computed  by  weighting 
each  multiple  correlation  by  its  degrees  of  freedom. 

For  the  nine  schools  that  had  both  FSG  and  practical  performance  criteria  available, 
comparative  summaries  were  prepared  combining  results  across  samples. 

Display  of  Selected  Test  x  Criterion  Results 

In  order  to  reduce  the  Type  I  error  associated  with  multiple  significance  tests,  incremental 
validities  for  individual  predictors  were  displayed  only  for  criteria  that  proved  significant  in  the 
full-model  regression  comparing  ASVAB  with  ASVAB  plus  six  ECAT  predictors. 

Results 


Combined  Results  Across  Samples 

Table  22  and  Table  23  show  the  mean  incremental  validities  and  combined  probabilities  for 
each  ECAT  predictor  for  the  School  Grade  criteria  and  the  Internal  School  Criteria.  The  two 
tables  are  remarkably  consistent  with  one  another.  In  both  tables,  the  accretion  probabilities  show 
that  every  ECAT  predictor  (except  for  Target  Identification  in  Table  23)  significantly  increases 
the  validity  of  ASVAB  by  itself. 

The  deletion  probability  is  a  measure  of  redundancy  with  other  ECAT  tests.  From  both  Table 
22  and  Table  23,  it  is  clear  that  either  Integrating  Details  or  Assembling  Objects  can  be  deleted 
without  significant  decrease  in  validity,  but  not  both,  because  the  Spatial  Composite  cannot  be 
deleted.  On  the  other  hand,  neither  working  memory  test  and  neither  tracking  test  can  be  deleted 
without  significant  decrease  in  validity.4  Spatial  Orientation  can  be  deleted. 

The  mean  increments  or  decrements  from  a  single  test  are  small.  All  ECAT  tests  show 
validity  increments  in  some  schools  that  are  much  larger  than  their  means,  as  will  be  shown  in  the 
next  section. 


4These  findings  for  the  spatial  ability  tests  and  working  memory  tests  confirm  those  of  Wolfe,  Alderton,  and 
Larson  (1993)  for  nine  Navy  schools. 
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Table  22 


Mean  Incremental  Validities  From  Adding  or  Deleting  one  ECAT  Test  to  the  ASVAB 
for  18  School  Grade  Criteria  ( N  =  8607) 


Simple 

Validity 

pale) 

Validity 

pale) 

Predictor 

Validity 

Rasvab+i 

Increase 

for  Accretion 

^  ASVAB+ECAT-l 

Decrease 

for  Deletion 

Mental  Counters 

.451 

.718 

.005 

4.86X10"16 

.728 

.001 

.007 

Sequential  Memory 

.410 

.717 

.004 

1.84xlO"10 

.728 

.001 

.011 

Integrating  Details 

.498 

.717 

.004 

5.46xl0“12 

.729 

.001 

.178 

Assembling  Objects 

.474 

.718 

.005 

1.47X10"12 

.728 

.001 

.089 

One-Hand  Tracking 

-.301 

.718 

.005 

1.97X10"8 

.728 

.001 

.039 

Two-Hand  Tracking 

-.339 

.719 

.006 

1.43X10'8 

.728 

.002 

.011 

Target  Identification 

-.257 

.716 

.003 

5.76X10"5 

.726 

.001 

.006 

Spatial  Orientation 

.475 

.111 

.004 

6.98X10'10 

.727 

.001 

.302 

Figural  Reasoning 

.499 

.111 

.004 

6.12X10-12 

.726 

.001 

.003 

Memory  Composite 

.477 

.719 

.006. 

<1.4  xlCT17 

.726 

.002 

.000 

Spatial  Composite 

.535 

.719 

.006 

<1.4  xlO-17 

.726 

.001 

.016 

Tracking  Composite 

-.342 

.719 

.006 

4.48  xlO"10 

.725 

.003 

.006 

Memory  Factor 

.492 

.714 

.006 

<1.4  xlO-17 

.720 

.001 

.010 

Space  Factor 

.590 

.717 

.009 

1.39X10"17 

.718 

.003 

Psychomotor  Factor 

-.378 

.714 

.007 

9.20X10-12 

.111 

.003 

.001 

Notes.  1.  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery. 

2.  The  first  six  rows  of  the  table  use  a  9-predictor  representation  of  the  ECAT  and  a  10-predictor  representation  of  the 
ASVAB.  The  second  set  of  six  rows  represent  the  ECAT  by  6  predictors  (3  tests  and  3  composites)  and  the  ASVAB 
by  10  predictors.  The  last  three  rows  represent  the  ECAT  by  3  factors  and  the  ASVAB  by  4  factors. 
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Table  23 


Mean  Incremental  Validities  From  Adding  or  Deleting  one  ECAT  Test  to  the  ASVAB 
for  19  Internal  School  Criteria  ( N  =  8490) 


Simple 

Validity 

^(X2g) 

Validity 

nxi) 

Predictor 

Validity 

^ASVAB+l 

Increase 

for  Accretion 

Rasvab+ecat-\ 

Decrease 

for  Deletion 

Mental  Counters 

.393 

.628 

.009 

2.37x10'" 

.650 

.003 

.010 

Sequential  Memory 

.354 

.626 

.007 

1.18XKT9 

.650 

.003 

.001 

Integrating  Details 

.427 

.624 

.005 

4.45x10^ 

.653 

.001 

.848 

Assembling  Objects 

.418 

.627 

.009 

1.57X10-9 

.651 

.002 

.155 

One-Hand  Tracking 

-.289 

.633 

.015 

1.71  xlO"9 

.650 

.003 

.016 

Two-Hand  Tracking 

-.301 

.632 

.014 

5.93xl0"9 

.649 

.004 

.001 

Target  Identification 

-.223 

.620 

.002 

6.32X10'2 

.650 

.002 

.136 

Spatial  Orientation 

.411 

.625 

.006 

1.21  xlO'5 

.650 

.001 

.300 

Figural  Reasoning 

.438 

.626 

.007 

7.12X10'9 

.649 

.002 

.049 

Memory  Composite 

.414 

.630 

.011 

1.54X10'14 

.645 

.006 

.000 

Spatial  Composite 

.465 

.629 

.010 

6.34X10'12 

.649 

.002 

.039 

Tracking  Composite 

-.315 

.635 

.016 

1.25xlO"10 

.639 

.012 

.000 

Memory  Factor 

.427 

.622 

.013 

<1.4  XlO'17 

.637 

.004 

.000 

Space  Factor 

.514 

.625 

.016 

<1.4  xlO'17 

.637 

.004 

.000 

Psychomotor  Factor 

-.344 

.626 

.016 

5.68xl0”12 

.630 

.011 

.000 

Notes.  1 .  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery. 

2.  The  first  six  rows  of  the  table  use  a  9-predictor  representation  of  the  ECAT  and  a  10-predictor  representation  of  the 
ASVAB.  The  second  set  of  six  rows  represent  the  ECAT  by  6  predictors  (3  tests  and  3  composites)  and  the  ASVAB 
by  10  predictors.  The  last  three  rows  represent  the  ECAT  by  3  factors  and  the  ASVAB  by  4  factors. 


Table  24  and  Table  25  show  the  similar  results  for  the  10  schools  that  had  both  FSG  and 
Performance  criteria  available.  The  results  are  somewhat  different  from  the  summaries  across  all 
18  schools.  For  the  FSG  criteria,  only  three  predictors  cannot  be  deleted  without  significant  loss: 
the  Memory  composite  and  the  Memory  and  Space  Factors.  Either  Mental  Counters  or  Sequential 
Memory  could  be  deleted,  but  not  both. 

The  performance  criteria  for  the  same  10  schools  show  much  larger  incremental  validities  and 
more  significant  effects.  In  Table  25,  the  Memory  and  Tracking  composites  plus  Sequential 
memory  and  both  tracking  tests  have  unique  predictive  power,  as  shown  by  their  significant 
deletion  probabilities.  The  strong  showing  for  the  psychomotor  tests  is  impressive,  considering 
that  the  sample  did  not  include  the  Army’s  11H  school,  which  showed  extremely  large  validity 
increments.  In  addition,  the  same  significant  finding  for  the  Memory  composite  that  was  observed 
for  the  FSG  criteria  also  applies  to  the  performance  criteria. 
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Table  24 


Mean  Incremental  Validities  From  Adding  or  Deleting  one  EC  AT  Test  to  the  ASVAB 
for  Final  School  Grade  Criteria  of  10  Schools  With  Dual  Criteria  ( N  =  3922) 


Predictor 

Simple 

Validity 

Rasvab+i 

Validity 

Increase 

p(&.) 

for  Accretion 

Rasvab+ecat-  1 

Validity 

Decrease 

^(X^) 

for  Deletion 

Mental  Counters 

.520 

.791 

.007 

9.54X10"14 

.795 

.002 

.071 

Sequential  Memory 

.489 

.789 

.006 

2.39X10-11 

.796 

.001 

.109 

Integrating  Details 

.550 

.789 

.005 

2.59xlO'10 

.796 

.001 

.253 

Assembling  Objects 

.523 

.788 

.005 

5.62x10^ 

.797 

.000 

.847 

One-Hand  Tracking 

-.325 

.786 

.003 

1.93X10"4 

.796 

.001 

.128 

Two-Hand  Tracking 

-.362 

.785 

.002 

3.01  xlO'3 

.797 

.000 

.387 

Target  Identification 

-.299 

.785 

.002 

5.36X10"2 

.796 

.001 

Spatial  Orientation 

.529 

.788 

.004 

5.97X10"9 

.796 

.001 

.325 

Figural  Reasoning 

.559 

.788 

.005 

1.34xlO-10 

.796 

.001 

Memory  Composite 

.559 

.792 

.009 

<1.4  xlO"17 

.794 

.002 

.000 

Spatial  Composite 

.591 

.791 

.008 

1.19X10-14 

.796 

.001 

.210 

Tracking  Composite 

-.367 

.786 

.003 

1.78x10^ 

.796 

.000 

Memory  Factor 

.575 

,788 

.010 

<1.4  xlO-17 

.790 

.001 

.011 

Space  Factor 

.659 

.790 

.013 

1.75xl0'n 

.789 

.003 

.000 

Psychomotor  Factor 

-.408 

.780 

.003 

2. 05x1 0’5 

.792 

.001 

.339 

Notes.  1.  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery. 

2.  The  first  six  rows  of  the  table  use  a  9-predictor  representation  of  the  ECAT  and  a  10-predictor  representation  of  the 
ASVAB.  The  second  set  of  six  rows  represent  the  ECAT  by  6  predictors  (3  tests  and  3  composites)  and  the  ASVAB 
by  10  predictors.  The  last  three  rows  rq)resent  the  ECAT  by  3  factors  and  the  ASVAB  by  4  factors. 
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Table  25 


Mean  Incremental  Validities  From  Adding  or  Deleting  one  ECAT  Test  to  the  ASVAB 
for  Performance  Criteria  of  10  Schools  With  Dual  Criteria  ( N  =  3828) 


Predictor 

Simple 

Validity 

R  ASVAB  + 1 

Validity 

Increase 

PQtn) 

for  Accretion 

Rasvab+ecat-i 

Validity 

Decrease 

P&h) 

for  Deletion 

Mental  Counters 

.466 

.657 

.019 

8.18xl0'12 

.675 

.005 

.057 

Sequential  Memory 

.441 

.653 

.015 

5.70x10"" 

.675 

.004 

.030 

Integrating  Details 

.459 

.648 

.010 

3.74X10"6 

.679 

.001 

.767 

Assembling  Objects 

.449 

.650 

.012 

1.96xl0"8 

.678 

.001 

.525 

One-Hand  Tracking 

-.309 

.651 

.013 

4.41  xlO"7 

.675 

.004 

.018 

Two-Hand  Tracking 

-.301 

.647 

.010 

5.21  xlO"6 

.674 

.005 

.008 

Target  Identification 

-.252 

.642 

.004 

4.17xl0~2 

.674 

.002 

.313 

Spatial  Orientation 

.435 

.646 

.008 

1.21  xlO"5 

.675 

.001 

.360 

Figural  Reasoning 

.469 

.650 

.012 

4.99  xlO"7 

.674 

.001 

.641 

Memory  Composite 

.502. 

.661 

.023 

3.33xl0“16 

.668 

.007 

.000 

Spatial  Composite 

.501 

.654 

.016 

4.25x10"" 

.673 

.002 

.135 

Tracking  Composite 

-.326 

.650 

.013 

2.66  xlO"7 

.668 

.007 

.000 

Memory  Factor 

.513 

.649 

.028 

<  1.4xl0"17 

.659 

.004 

.001 

Space  Factor 

.559 

.652 

.030 

<  1.4xl0“17 

.658 

.005 

.000 

Psychomotor  Factor 

-.358 

.635 

.014 

2.04x1 0"8 

.656 

.008 

.000 

Notes.  1.  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery. 

2.  The  first  six  rows  of  the  table  use  a  9-predictor  representation  of  the  ECAT  and  a  10-predictor  representation  of  the 
ASVAB.  The  second  set  of  six  rows  represent  the  ECAT  by  6  predictors  (3  tests  and  3  composites)  and  the  ASVAB 
by  10  predictors.  The  last  three  rows  represent  the  ECAT  by  3  factors  and  the  ASVAB  by  4  factors. 


Test  x  Criterion  Results 

Validity  increments  for  adding  just  one  new  predictor  to  the  ASVAB  are  shown  for  each 
predictor  and  each  significant  criterion  in  Table  26  and  Table  28  for  the  School  Grade  criteria  and 
in  Table  27  and  Table  29  for  the  Internal  School  criteria.  (Table  1-4  of  Appendix  I  gives  the 
results  for  all  significant  criteria  for  all  schools.) 

There  are  a  large  number  of  significant  findings  shown  in  the  tables.  We  mention  below  those 
significant  incremental  validities  greater  than  .02.  Values  larger  than  .04  are  listed  in  parentheses. 
This  does  not  imply  that  the  lesser  values  are  not  important,  however. 

Memory  Factor:  11H(B)  EVTSUM,  AE  FSG,  APS  typing  speed  (AFPT70)  (.05),  ATC 
Basic  Approach  Control  Operations  (.09  and  .06),  Navy  AC  performance  (.15),  AE 
performance  (SUM2),  and  OS  performance^ 

Psychomotor  Factor:  11H  all  criteria  (as  high  as  .178),  ATC  Basic  Approach  Control 
Operations  (.05). 
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Table  26 


Incremental  Validities  From  Adding  one  ECAT  Factor  to  Four  ASVAB  Factors 
for  Significant  School  Grades  From  Full  Model 


School 

Criterion 

Memory 

Psychomotor 

Space 

11H(A)5 

EVTSUM 

.012* 

.034** 

.027** 

11H(B)5 

EVTSUM 

.021** 

.086** 

.023** 

13F1 

FSG 

* 

* 

CO 

o 

.007** 

.028** 

APS1 

FSG 

.009** 

.000 

.006** 

ATC1 

FSG 

.012** 

.006* 

.013** 

AC1 

FSG 

.000 

.000 

.000 

AE1 

FSG 

.024** 

.003* 

.022** 

AVI 

FSG 

.005** 

.001 

.004** 

FC1 

FSG 

.000 

.000 

.003** 

MM1 

FSG 

.000 

.000 

.006** 

OS1 

FSG 

.007** 

.000 

.008** 

RM1 

FSG 

.005* 

.001 

.004 

Notes.  1.  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery, 
FSG  =  Final  School  Grade. 

2.  For  definitions  of  schools  and  criteria,  see  Tables  12-14. 

*  p  <  .05  for  uncorrected  R  increase.  **  p  <  .01  for  uncorrected  R  increase. 


Table  27 

Incremental  Validities  From  Adding  one  ECAT  Factor  to  Four  ASVAB  Factors 
for  Significant  Internal  Criteria  From  Full  Model 


School 

Criterion 

Memory 

Psychomotor 

Space 

11H(A)6 

TO_l 

.000 

.055** 

.003 

11H(B)9 

ITVTOW 

.000 

.178** 

.039** 

13F3 

FIRING 

Oil** 

.005** 

009** 

APS3 

AFPT70 

.051** 

.015* 

.034** 

ATC(A)4 

BLK5A 

.089* 

.047* 

.120** 

ATC(B)4 

BLK5A 

.060** 

.053** 

.078** 

AC2 

PERF 

.150* 

.019 

.142 

AE2 

SUM2 

.024** 

.000 

.013** 

AV4 

PERFORM 

.009 

.014* 

.011* 

FC2 

RADAR 

.002* 

.004 

.000 

MM1 

FSG 

.000 

.000 

.006** 

OS3 

PERF 

.020** 

.008** 

.025** 

Notes.  1.  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery, 
FSG  =  Final  School  Grade. 

2.  For  definitions  of  schools  and  criteria,  see  Tables  12-14. 

*  p  <  .05  for  uncorrected  R  increase.  **  p  <  .01  for  uncorrected  R  increase. 
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Table  28 


Incremental  Validities  From  Adding  one  ECAT  Test  to  the  ASVAB 
for  Significant  School  Grade  Criteria 


School 

Criterion 

Mental 

Counters 

Sequential 

Memory 

Integrating 

Details 

Assembling 

Objects 

11H(A)5 

EVTSUM 

.013* 

.008 

.004 

.027** 

11H(B)5 

EVTSUM 

.008 

.024* 

.016* 

.003 

13F1 

FSG 

.010** 

.009** 

.012** 

.012** 

APS1 

FSG 

.002 

.006** 

.003* 

.000 

ATC1 

FSG 

.015** 

.004 

.001 

.005* 

AE1 

FSG 

.010** 

.020** 

.019** 

.009* 

AVI 

FSG 

.007** 

.002* 

.002* 

.002* 

FC1 

FSQ 

.000 

.000 

.001 

.003** 

MM1 

FSG 

.000 

.000 

.003 

.009** 

OS1 

FSG 

.007** 

.003* 

.002* 

.002* 

RM1 

FSG 

.004 

.002 

.004 

.000 

One-Hand 

Two-Hand 

Target 

Spatial 

School 

Criterion 

Tracking 

Tracking 

Identification 

Orientation 

11H(A)5 

EVTSUM 

.019** 

.029** 

.011* 

.010* 

11H(B)5 

EVTSUM 

.059** 

.078** 

.021* 

.022** 

13F1 

FSG 

.005** 

.003** 

.003* 

.010** 

APS1 

FSG 

.000 

.000 

.000 

.002* 

ATC1 

FSG 

.006* 

.003 

.008** 

.010** 

AE1  • 

FSG 

.004* 

.000 

.004 

.004* 

AVI 

FSG 

.000 

.002 

.000 

.001 

FC1 

FSG 

.000 

.001* 

.001* 

.000 

MM1 

FSG 

.003* 

.000 

.000 

.000 

OS1 

FSG 

.000 

.001 

.000 

.003* 

RM1 

FSG 

.002 

.000 

on** 

.002 

Memory 

Spatial 

Tracking 

Figural 

School 

Criterion 

Composite 

Composite 

Composite 

Reasoning 

11H(A)5 

EVTSUM 

.015* 

.020** 

.028** 

.000 

11H(B)5 

EVTSUM 

.023* 

.013 

.080** 

.001 

13F1 

FSG 

.013** 

.017** 

.005** 

.000 

APS1 

FSG 

.006** 

.002* 

.000 

.010** 

ATC1 

FSG 

.013** 

.004* 

.005* 

.010** 

AE1 

FSG 

.021** 

020** 

.003* 

.002 

AVI 

FSG 

.007** 

.003* 

.001 

.001 

FC1 

FSG 

.000 

.003** 

.000 

.015 

MM1 

FSG 

.000 

.008** 

.000 

.002* 

OS1 

FSG 

.007** 

.003** 

.000 

.009** 

RM1 

FSG 

.005 

.002 

.002 

.004** 

Notes.  L  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery, 
FSG  =  Final  School  Grade. 

2.  For  definitions  of  schools  and  criteria,  see  Tables  12-14. 

*  p  <  .05  for  uncorrected  R  increase.  **  p  <  .01  for  uncorrected  R  increase. 
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Table  29 


Incremental  Validities  From  Adding  one  ECAT  Test  to  the  ASVAB 
for  Significant  Internal  School  Criteria 


Criterion 

Mental 

Counters 

Sequential 

Memory 

Integrating 

Details 

Assembling 

Objects 

11H(A)6 

TO_l 

.000 

.000 

.000 

.000 

11H(B)9 

ITVTOW 

.000 

.000 

.006 

.056* 

13F3 

FIRING 

.002* 

.007** 

.002* 

.002* 

APS3 

AFPT70 

.018** 

.034** 

.025** 

.010* 

ATC(A)4 

BLK5A 

.in** 

.006 

.026* 

.015 

ATC(B)4 

BLK5A 

.060* 

.032 

.014 

.040* 

AC2 

PERF 

.048 

.135* 

.045 

.126* 

AE2 

SUM2 

.008* 

.018** 

.005 

.004 

FC2 

RADAR 

.000 

.005** 

.000 

.001 

MM1 

FSG 

.000 

.000 

.003 

009** 

OS3 

PERF 

.017** 

.011** 

.006** 

.010** 

RM2 

PHASE3 

.004 

.000 

.002 

.000 

School 

Criterion 

One-Hand 

Tracking 

Two-Hand 

Tracking 

Target 

Identification 

Spatial 

Orientation 

11H(A)6 

TO_l 

.036** 

.044** 

.000 

.008 

11H(B)9 

ITVTOW 

.159** 

.172** 

.000 

.047* 

13F3 

FIRING 

.006** 

.002* 

.002 

.002* 

APS3 

AFPT70 

.006 

.028** 

.000 

.004 

ATC(A)4 

BLK5A 

.030 

.015 

.005 

.000 

ATC(B)4 

BLK5A 

.049** 

.034** 

.023* 

.044** 

AC2 

PERF 

.063 

.000 

.000 

.033 

AE2 

SUM2 

.000 

.000 

.009* 

.000 

FC2 

RADAR 

.002 

.004* 

.000 

.000 

MM1 

FSG 

.003* 

.000 

.000 

.000 

OS3 

PERF 

.003* 

.006** 

.000 

on** 

RM2 

PHASE3 

.000 

.000 

.006 

.006 

School 

Criterion 

Memory 

Composite 

Spatial 

Composite 

Tracking 

Composite 

Figural 

Reasoning 

11H(A)6 

TO_l 

.000 

.000 

.047** 

.007 

11H(B)9 

ITVTOW 

.004 

.047** 

.185** 

.000 

13F3 

FIRING 

.006** 

.003** 

.005** 

.003** 

APS3 

AFPT70 

.036** 

.024** 

.018** 

.014** 

ATC(A)4 

BLK5A 

.066** 

.031** 

.027 

.060** 

ATC(B)4 

BLK5A 

.063* 

.038* 

.049** 

.036 

AC2 

PERF 

.128 

.123 

.025 

.070 

AE2 

SUM2 

.019** 

.007 

.000 

.003 

FC2 

RADAR 

.003* 

.000 

.004* 

.003 

MM1 

FSG 

.000 

.008** 

.000 

.009** 

OS3 

PERF 

.019** 

.012** 

.005** 

.007** 

RM2 

PHASE3 

.003 

.002 

.000 

.000 

Notes.  1.  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery, 
FSG  =  Final  School  Grade. 

2.  For  definitions  of  schools  and  criteria,  see  Tables  12-14. 

*  p  <  .05  for  uncorrected  R  increase.  **  p  <  .01  for  uncorrected  R  increase. 
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Space  Factor:  11H  EVTSUM  and  ITVTOW.  13F  FSG,  APS  Typing  Speed,  AE  FSG, 
ATC  Basic  Approach  Control  Operations  (.12  and  .08),  and  OS  performance. 

Mental  Counters:  ATC  Basic  Approach  Control  Operations  (.11). 

Sequential  Memory:  11H  EVTSUM,  Aviation  Electrician  FSG,  APS  typing  speed,  and 
Navy  Air  Traffic  Control  performance  (.14). 

Integrating  Details:  APS  Typing  Speed  and  ATC  Basic  Approach  Control  Operations 
(.052). 

Assembling  Objects:  11H  TOW  Simulator  Tracking  (ITVTOW  .06),  ATC  Basic 
Approach  Control  Operations  (.04),  and  Navy  Air  Traffic  Control  performance  (.13). 

One-Hand  Tracking  11H  TOW  Firing  (EVTSUM  .06),  TOW  Simulator  Tracking  (TO_l 
and  ITVTOW  .16);  ATC  Basic  Approach  Control  Operations  (.05). 

Two-Hand  Tracking:  11H  TOW  Firing  (EVTSUM  .08),  TOW  Simulator  Tracking 
(TO_l  .04  and  ITVTOW  .17),  APS  typing  speed,  ATC  Basic  Approach  Control  Operations. 

Target  Identification:  ATC  Basic  Approach  Control  Operations,  and  11H  TOW  Firing 
EVTSUM. 

Spatial  Orientation:  ATC  Basic  Approach  Control  Operations  (.04),  11H  TOW  Firing 
EVTSUM  and  ITVTOW. 

Figural  Reasoning:  ATC  Basic  Approach  Control  Operations  (.06). 

It  is  interesting  that  Working  Memory  seems  to  predict  typing  speed  better  than  Tracking 
does  (.036  vs.  .018). 

It  is  evident  that  even  a  single  test  added  to  the  ASVAB  can  produce  large  validity  gains  for 
some  criteria,  with  the  largest  gains  exceeding  .10  from  the  two  tracking  tests,  the  two  memory 
tests,  and  Assembling  Objects. 
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Stepwise  Meta-analysis 


Method 

The  last  section  showed  the  mean  validity  changes  resulting  from  adding  or  deleting  a  single 
predictor  from  the  battery.  This  information  could  be  used  to  select  the  best  predictor  to  add  or 
delete  from  the  battery  in  order  to  maximize  validity  averaged  across  samples.  Suppose  we  were 
to  add  or  delete  that  predictor,  and  then  re-do  the  whole  analysis  in  each  sample,  then  average 
across  samples  to  determine  the  mean  incremental  validities  of  the  remaining  predictors  with 
respect  to  the  modified  battery.  Repeat  the  process  in  order  to  determine  the  next  predictor  to 
add  or  delete.  This,  in  essence,  is  what  we  mean  by  a  stepwise  meta-analysis. 

The  algorithm  is  quite  simple.  For  accretion,  select  the  predictor  with  the  greatest  mean 
incremental  validity.  Add  it  to  the  battery.  In  each  sample,  compute  the  incremental  validities  of 
each  of  the  remaining  unused  predictors  with  respect  to  the  modified  battery.  Average  across 
samples  to  compute  the  mean  incremental  validities  of  each  of  the  remaining  unused  predictors 
with  respect  to  the  modified  battery.  Select  the  predictor  with  the  greatest  mean  incremental 
validity,  and  repeat  the  procedure  until  all  predictors  have  entered  the  battery.  For  deletion,  find 
the  predictor  that  decreases  mean  validity  the  least,  delete  it  from  the  battery,  then  compute  the 
mean  validity  decrements  from  deleting  each  of  the  remaining  predictors  from  the  modified 
battery.  Repeat  the  procedure  until  all  predictors  have  been  removed  from  the  battery.5 

Appendix  K  gives  a  concrete  representation  of  this  procedure  in  the  form  of  a  SAS  program 
to  carry  out  both  accretion  and  deletion  meta-analyses.  All  of  the  results  in  this  section  were 
produced  by  this  program. 

Three  types  of  meta-analyses  were  done. 

1.  ASVAB  Kernel.  The  first  type  of  analysis  assumed  that  all  10  ASVAB  tests  remained  in 
regression  at  all  times,  and  concentrated  on  adding  or  deleting  only  the  ECAT  variables.  Four 
analyses  were  done  with  different  subsets  of  the  ECAT  variables: 

a.  All  nine  ECAT  tests. 

b.  Six  ECAT  tests  that  did  not  use  the  psychomotor  response  pedestal. 

c.  Three  P&P  ECAT  tests  only. 

d.  Several  combinations  of  three  ECAT  tests. 


5 A  similar  procedure  was  independently  developed  by  Abrahams  and  Alf  and  used  in  the  ECAT  analyses 
done  by  Abrahams,  Pass,  Kusulas,  Cole,  and  Kieckhaefer  (1993).  Both  procedures  bear  a  strong  similarity  to  Horst 
(1955),  who  showed  how  to  maximize  the  mean  squared  multiple  correlation. 
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These  analyses  made  it  possible  to  determine  the  incremental  value  of  P&P  tests, 
computerized  cognitive  tests,  and  computerized  psychomotor  tests,  respectively. 

2.  AFQT  Kernel.  The  second  type  of  analysis  assumed  that  only  the  four  AFQT  tests  (WK,  PC, 
AR,  MK)  remained  in  regression  at  all  times,  and  allowed  either  an  ECAT  test  or  another 
AS  VAB  test  to  enter  or  exit  the  regression  equations. 

3.  Null  Kernel.  The  third  type  of  analysis  allowed  even  the  AFQT  tests  to  be  replaced  by  ECAT 
tests  if  the  result  were  higher  multiple  correlations. 

All  analyses  used  fully  corrected  mean  validities,  i.e.,  range-corrected,  then  Wherry-shrunken, 
then  corrected  for  criterion  reliability,  then  averaged  by  weighting  them  by  their  degrees  of 
freedom. 

Results 

Deletion  generally  produced  multiple  correlations  equal  to  or  larger  than  accretion  in  the 
range  of  10-12  predictors,  so  only  the  deletion  results  are  shown. 

The  three  analyses  with  all  nine  ECAT  tests  are  presented  in  Table  30,  Table  31,  and  Table 
32.  Certain  ECAT  tests  seem  to  enter  regression  early:  Two-Hand  Tracking,  Mental  Counters, 
and  Assembling  Objects.  The  first  ASVAB  tests  to  be  displaced  are  Numerical  Operations, 
Mechanical  Comprehension,  and  General  Science. 

The  substitutional  validity  of  the  ECAT  tests  can  be  determined  by  comparing  the  validities 
for  10-test  batteries.  From  Table  31,  line  10,  it  is  easy  to  see  that  replacing  NO,  MC,  and  GS  with 
Assembling  Objects,  Two-Hand  tracking,  and  one  of  the  Working  Memory  tests  increases  the 
mean  battery  validity  for  predicting  School  Grades  by  1.4%  and  internal  criteria  by  3.6%.  These 
gains  can  be  achieved  without  changing  the  AFQT  or  increasing  total  testing  time. 

Figure  11  depicts  the  results  from  Table  30  and  Table  31.  When  the  regression  equations 
start  with  all  10  ASVAB  tests,  the  validity  curve  tends  to  level  out  after  three  ECAT  tests  are 
entered.  These  three  ECAT  tests  correspond  to  the  three  underlying  factors  in  the  ECAT  battery: 
Psychomotor  Ability,  Working  Memory,  and  Spatial  Ability.  Three  ECAT  tests  produce  76%  of 
the  gain  from  using  the  full  battery  of  nine  tests  for  predicting  Internal  Criteria.  The  two  curves 
on  the  left  show  what  happens  when  the  four  tests  in  the  AFQT  are  used  as  the  starting  point,  and 
other  ASVAB  or  ECAT  tests  are  free  to  enter.  With  seven  or  eight  tests,  the  curve  rises  above  the 
validity  line  for  the  10-test  ASVAB. 
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Table  30 


Means  of  Fully  Corrected  Multiple  Correlations  for 
Stepwise  Deletion  Meta-analysis  Assuming  10  ASVAB  Tests  in  Model 


Number  of 
Predictors 

School  Grade  Criteria 

Internal  Criteria 

Predictor 

R 

Predictor 

R 

10 

10  ASVAB  Tests 

.714 

10  ASVAB  Tests 

.620 

11 

+  Two-Hand  Tracking 

.719 

+  Two-Hand  Tracking 

.633 

12 

+  Mental  Counters 

.723 

+  Mental  Counters 

.642 

13 

+  Assembling  Objects 

.725 

+  Figural  Reasoning 

.646 

14 

+  Figural  Reasoning 

.727 

+  One-Hand  Tracking 

.648 

15 

+  One-Hand  Tracking 

.728 

+  Sequential  Memory 

.651 

16 

+  Sequential  Memory 

.729 

+  Assembling  Objects 

.652 

17 

+  Target  Identification 

.730 

+  Target  Identification 

.653 

18 

+  Spatial  Orientation 

.730 

+  Spatial  Orientation 

.654 

19 

+  Integrating  Details 

.730 

+  Integrating  Details 

.653 

Note.  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery. 


Table  31 

Means  of  Fully  Corrected  Multiple  Correlations  for 
Stepwise  Deletion  Meta-analysis  Assuming  Four  AFQT  Tests  in  Model 


Number  of 
Predictors 

School  Grade  Criteria 

Internal  Criteria 

Predictor 

R 

Predictor 

R 

4 

4  AFQT  Tests 

.671 

4  AFQT  Tests 

.565 

5 

+  Auto-Shop  Information 

.697 

+  Auto-Shop  Information 

.594 

6 

+  Assembling  Objects 

.706 

+  Two-Hand  Tracking 

.613 

7 

+  Two-Hand  Tracking 

.711 

+  Mental  Counters 

.624 

8 

+  Coding  Speed 

.716 

+  Coding  Speed 

.632 

9 

+  Electronics  Information 

.721 

+  Electronics  Information 

.638 

10 

+  Mental  Counters 

.724 

+  Assembling  Objects 

.642 

11 

+  Figural  Reasoning 

.725 

+  One-Hand  Tracking 

.645 

12 

+  General  Science 

.726 

+  Sequential  Memory 

.647 

13 

+  One-Hand  Tracking 

.727 

+  Figural  Reasoning 

.649 

14 

+  Sequential  Memory 

.728 

+  General  Science 

.651 

15 

+  Target  Identification 

.729 

+  Mechanical  Comprehension 

.652 

16 

+  Spatial  Orientation 

.730 

+  Target  Identification 

.653 

17 

+  Integrating  Details 

.730 

+  Numerical  Operations 

.653 

18 

+  Numerical  Operations 

.730 

+  Spatial  Orientation 

.654 

19 

+  Mechanical  Comprehension 

.730 

+  Integrating  Details 

.653 

Note.  AFQT  =  Armed  Forces  Qualification  Test. 
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Table  32 


Means  of  Fully  Corrected  Multiple  Correlations  for 
Stepwise  Deletion  Meta-analysis  With  or  Without  ASVAB  Tests  in  Model 


Number  of 

Predictors 

School  Grade  Criteria 

Internal  Criteria 

Predictor 

R 

Predictor 

R 

1 

Mathematics  Knowledge 

.563 

Arithmetic  Reasoning 

.517 

2 

+  Auto-Shop  Information 

.648 

+  Auto-Shop  Information 

.553 

3 

+  Arithmetic  Reasoning 

.679 

+  Mathematics  Knowledge 

.582 

4 

+  Paragraph  Comprehension 

.694 

+  Two-Hand  Tracking 

.601 

5 

+  Assembling  Objects 

.703 

+  Coding  Speed 

.614 

6 

+  Electronics  Information 

.709 

+  Mental  Counters 

.624 

7 

+  Coding  Speed 

.715 

+  Electronics  Information 

.631 

8 

+  Two-Hand  Tracking 

.719 

+  Paragraph  Comprehension 

.636 

9 

+  Mental  Counters 

.722 

+  Assembling  Objects 

.639 

10 

+  Word  Knowledge 

.724 

+  One-Hand  Tracking 

.642 

11 

+  Figural  Reasoning 

.725 

+  Word  Knowledge 

.645 

12 

+  General  Science 

.726 

+  Sequential  Memory 

.647 

13 

+  One-Hand  Tracking 

.727 

+  Figural  Reasoning 

.649 

14 

+  Sequential  Memory 

.728 

+  General  Science 

.651 

15 

+  Target  Identification 

.729 

+  Mechanical  Comprehension 

.652 

16 

+  Spatial  Orientation 

.730 

+  Target  Identification 

.653 

17 

+  Integrating  Details 

.730 

+  Numerical  Operations 

.653 

18 

+  Numerical  Operations 

.730 

+  Spatial  Orientation 

.654 

19 

+  Mechanical  Comprehension 

.730 

+  Integrating  Details 

.653 

Note.  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery. 
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Mean  Multiple  Correlation  Increment 


Number  of  Predictors 


Figure  11.  Incremental  validity  as  a  function  of  number  of  ECAT  tests. 

In  all  three  analyses,  the  same  three  ECAT  tests  emerge  first:  Two-Hand  Tracking,  Mental 
Counters,  and  Assembling  Objects.  Each  of  these  tests  represents  a  different  ability  factor.  How 
sensitive  are  the  results  to  the  choice  of  the  particular  test  representing  a  factor?  Table  33  shows 
the  validities  of  several  alternative  test  combinations.  To  three  decimal  places,  the  validity  is  the 
same  whether  One-Hand  or  Two-Hand  Tracking  is  chosen  to  represent  Psychomotor  Ability. 
Using  Integrating  Details  instead  of  Assembling  Objects,  or  Sequential  Memory  instead  of  Mental 
Counters  decreases  validity  for  Internal  Criteria  by  only  .001  and  leaves  School  Grade  prediction 
unchanged.  The  decrease  is  twice  as  large  if  Figural  Reasoning  is  used  instead  of  Mental 
Counters,  but  is  still  quite  small.  It  seems  that  the  important  thing  is  to  include  good  measures  of 
the  three  ECAT  factors.  The  particular  test  chosen  may  be  largely  a  matter  of  chance. 


Table  33 


Means  of  Fully  Corrected  Multiple  Correlations  for  Alternative 
Combinations  of  Three  ECAT  Tests  Assuming  10  ASVAB  Tests  in  Model 


School  Internal 

Grade  Criteria 

Criteria 

_ Predictors _ _ R 

10  ASVAB  Tests  .714  .620 

+  Two-Hand  Tracking  +  Mental  Counters  +  Assembling  Objects  .725  .645 

+  One-Hand  Tracking  +  Mental  Counters  +  Assembling  Objects  .725  .645 

+  Two-Hand  Tracking  +  Mental  Counters  +  Integrating  Details  .725  .644 

+  Two-Hand  Tracking  +  Sequential  Memory  +  Assembling  Objects  .725  .644 

+  Two-Hand  Tracking  +  Figural  Reasoning  +  Assembling  Objects _ .725 _ .643 


Note.  ECAT  =  Enhanced  Computer  Administered  Testing,  ASVAB  =  Armed  Services  Vocational  Aptitude  Battery. 


Turning  to  considerations  of  practicability,  the  ECAT  tests  can  be  classified  according  to  ease 
of  implementation.  The  three  (formerly)  P&P  tests.  Assembling  Objects,  Figural  Reasoning,  and 
Spatial  Orientation,  are  easiest  to  implement.  The  three  computerized  cognitive  tests.  Mental 
Counters,  Sequential  memory,  and  Integrating  Details,  require  availability  of  computers  for 
administration.  The  psychomotor  tests  require  not  only  a  computer,  but  also  a  special  Response 
Pedestal,  making  them  the  most  expensive  to  implement  and  maintain. 

How  much  of  the  incremental  validity  of  ECAT  is  due  to  the  psychomotor  tests?  Table  34 
shows  the  results  of  a  stepwise  analysis  with  the  psychomotor  tests  excluded.  The  last  line  of  the 
table  should  be  compared  with  the  last  line  of  Table  30.  Without  psychomotor  tests,  ECAT 
increases  validity  only  .012  instead  of  .016  for  School  grades  and  only  .019  instead  of  .033  for 
Internal  Criteria.  Thus  25%  to  42%  of  ECAT’s  incremental  validity  comes  from  the  psychomotor 
tests  (principally  Two-Hand  Tracking),  even  if  they  are  entered  last  into  regression. 


Table  34 

Means  of  Fully  Corrected  Multiple  Correlations  for 
Stepwise  Accretion  Meta-analysis  Without  Psychomotor  Tests  in  Model 


Number  of 
Predictors 

School  Grade  Criteria 

Internal  Criteria 

Predictor 

R 

Predictor 

R 

10 

10  ASVAB  Tests 

.714 

10  ASVAB  Tests 

.620 

11 

+  Mental  Counters 

.719 

+  Mental  Counters 

.629 

12 

+  Assembling  Objects 

.722 

+  Assembling  Objects 

.634 

13 

+  Figural  Reasoning 

.723 

+  Figural  Reasoning 

.637 

14 

+  Sequential  Memory 

.724 

+  Sequential  Memory 

.638 

15 

+  Spatial  Orientation 

.725 

+  Spatial  Orientation 

.640 

16 

+  Integrating  Details 

.726 

+  Integrating  Details 

.639 
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Suppose  the  Spatial  Orientation  test  and  all  tests  which  require  computer  administration  were 
omitted  from  the  battery,  leaving  only  Assembling  Objects  and  Figural  Reasoning.  Table  35  shows 
that  validity  increases  .009  for  predicting  Grades  and  .014  for  predicting  Internal  Criteria,  over  the 
ASVAB  alone.  Thus  these  P&P  tests  can  account  for  56%  and  42%  of  the  ECAT  incremental 
validity  for  predicting  Grades  or  the  Internal  Criteria,  respectively,  if  they  are  entered  first  into 
regression. 

Table  35 

Means  of  Fully  Corrected  Multiple  Correlations  for 
Stepwise  Deletion  Meta-analysis  Without  Computerized  Tests  in  Model 


Number  of 
Predictors 

School  Grade  Criteria 

Internal  Criteria 

Predictor 

R 

Predictor 

R 

10 

10  ASVAB  Tests 

.714 

10  ASVAB  Tests 

.620 

11 

+  Assembling  Objects 

.719 

+  Assembling  Objects 

.628 

12 

+  Spatial  Orientation 

.721 

+  Figural  Reasoning 

.632 

13 

+  Figural  Reasoning 

.723 

+  Spatial  Orientation 

.634 

These  relationships  are  depicted  in  Figure  12.  It  is  clear  that,  once  the  P&P  tests  are  forced 
into  regression  first,  the  other  computerized  cognitive  tests  produce  very  little  further 
improvement.  However,  even  after  all  cognitive  tests  are  forced  into  regression,  the  psychomotor 
tests  have  substantial  predictive  power.  Nevertheless,  about  half  of  the  criterion  variance  is  still 
unaccounted  for. 
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Criterion  Variance 


Predicted  and  Unpredicted  Criterion  Variance 


Figure  12.  Prediction  variance  from  paper-and-pencil,  computerized  cognitive,  and  psychomotor 
tests. 
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Discussion 


It  appears  that  chance  plays  a  large  role  in  the  order  of  accretion  or  deletion.  For  example, 
Integrating  Details  was  the  first  ECAT  test  deleted  in  all  of  the  analyses,  while  Assembling 
Objects  remained  until  only  three  ECAT  tests  were  left.  Yet,  as  the  previous  section  has  shown, 
there  is  no  significant  difference  in  the  incremental  validities  for  these  two  tests.  The 
nonsignificant y  lower  incremental  validity  for  Integrating  Details  caused  it  to  be  removed  first. 
Since  the  two  tests  measure  similar  constructs,  and  since  spatial  ability  is  essential,  removing  one 
of  the  tests  causes  the  other  one  to  assume  an  important  position.  Several  other  caveats  are  in 
order: 

1.  The  battery  that  maximizes  mean  validity  may  not  maximize  differential  prediction,  which 
is  important  when  the  tests  are  used  for  classification. 

2.  Some  tests  require  much  less  time  or  equipment  to  administer  than  others.  Optimal  battery 
construction  should  take  these  costs  into  account. 

3.  The  mean  validities  are  the  averages  over  a  set  of  schools  that  are  not  a  representative, 
random  sample  of  all  military  training  schools. 

4.  Minor  changes  in  the  set  of  schools  or  criteria  can  cause  large  changes  in  the  order  with 
which  predictors  enter  or  leave  the  equations. 

5.  It  may  not  be  a  good  idea  to  select  tests  based  on  averages  that  include  schools  where  the 
incremental  validities  are  so  small  that  the  test  will  never  be  used  there.  Instead,  tests  may  be 
selected  for  their  maximum  incremental  validities,  or  the  frequency  with  which  their  validities 
exceed  a  certain  threshold  for  inclusion  in  selection  equations. 

6.  It  is  unlikely  that  least-squares  multiple  regression  weights  will  be  used  for  selection  or 
classification.  Current  practice  is  to  use  unit  or  low  integer  weights. 
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Validities  of  Unit- weighted  Composites 


Introduction 

All  of  the  results  up  to  now  have  relied  on  least-squares  linear  multiple  regression.  In  the  ASVAB’s 
operational  use,  integer- weighted  composites  of  tests  are  used  for  selection  and  classification  for  two  very 
good  reasons: 

1.  On  small  samples,  regression  equations  do  not  cross- validate  as  well  as  integer- weighted 
composites. 

2.  Full  least-squares  regression  equations  often  involve  negative  weights,  which,  in  effect, 
penalize  examinees  for  doing  well  on  those  tests  with  negative  weights. 

Additionally,  the  regression  analysis  of  incremental  validity  has  been  criticized  as  being  too 
conservative.  By  optimally  weighting  all  10  ASVAB  tests  as  a  basis  for  comparison,  it  is  argued, 
the  ASVAB’s  validity  has  been  inflated  to  unrealistic  levels,6  leaving  little  room  for  new  tests  to 
improve  the  validity. 

The  recommended  alternative  would  be  to  estimate  the  incremental  validity  of  new  tests  over 
the  existing  selector  composite. 

It  is  therefore  desirable  to  examine  the  use  of  ECAT  tests  under  conditions  similar  to  the  way 
the  ASVAB  is  used  now. 

Method 

Programs  were  developed  to  compute  the  validity  of  any  given  unit-weighted  composite,  and 
then  search  for  the  best  test  to  add  (with  unit  weights)  to  a  composite  to  maximize  validity.  All 
correlations  were  corrected  for  range-restriction,  but  not  for  criterion  unreliability.  The  programs 
were  first  applied  to  the  existing  selector  composites  to  find  which  ECAT  test  would  increase 
validity  the  most  for  a  given  school  and  find  which  additional  ASVAB  test  would  increase  the 
validity  the  most.  The  purpose  was  to  determine  whether  adding  an  ECAT  test  to  a  selector 
composite  would  increase  validity  more  than  adding  an  ASVAB  test.  In  addition,  since 
Assembling  Objects  appeared  to  be  a  particularly  promising  test,  it  was  evaluated  for  possible 
addition  to  each  operational  selector  composite. 

For  comparison,  optimal  regression  equations  were  computed  for  each  set  of  variables  that 
had  appeared  in  the  unit-weighted  composites. 


6However,  the  Wherry  correction  should  shrink  these  estimates  to  their  population  values.  Part  of  our  purpose 
is  to  show  that  these  criticisms  are  not  valid,  and  that,  on  the  contrary,  the  existing  selector  composites  greatly 
underestimate  the  ASVAB's  validity,  resulting  in  incorrect  estimates  of  incremental  validity. 
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Results 


The  range-corrected  validities  of  the  additions  to  the  operational  selector  composites  are 
shown  in  Table  36  for  Final  School  Grades  and  in  Table  37  for  Internal  School  Criteria.  The 
three  right-most  columns  give  the  validities  of  the  integer-weighted  composites  and  their 
differences.  The  first  column  of  the  three,  labeled  “Comp”  gives  the  validity  of  the  composite 
defined  at  the  far  left  of  the  page.  The  second  column,  “Comp  -  OP,”  gives  the  difference  between 
the  first  column  and  the  validity  of  the  operational  composite.  The  third  column,  “Comp  - 
(OP+ASVAB),”  gives  the  difference  between  the  first  column  and  the  validity  of  the  operational 
composite  plus  the  best  ASVAB  test  to  add  to  it.  This  column  is  the  best  comparison  of  ECAT 
with  ASVAB. 

With  the  exceptions  of  AC,  EM,  and  EN  Final  School  Grades,  an  ECAT  test  can  be  added  to 
the  operational  composite  to  improve  validity.  However,  in  about  half  the  cases,  an  ASVAB  test 
will  improve  validity  even  more  than  an  ECAT  test.  Significant  cases  where  an  ECAT  test  has 
much  greater  incremental  validity  than  any  ASVAB  test  are  the  11H  criteria  using  Two-Hand 
Tracking,  APS  Typing  Speed  using  Sequential  memory,  and  ATC  Basic  Ground  Approach 
Control  using  Mental  Counters, 

Assembling  Objects  is  seldom  better  than  an  ASVAB  test  for  improving  an  operational 
selector  composite  if  added  with  unit  weight  except  for  the  11H  and  ATC  criteria,  which  are 
inconsistent  between  the  alternate  curricula  for  those  schools.. 

The  columns  on  the  left  give  the  results  of  the  regression  analyses  with  the  same  variables.  In 
many  cases,  the  integer  weights  have  validities  about  the  same  as  the  regression’s  multiple 
correlation.  However,  the  differences  are  large  for  11H,  APS,  AV,  EN,  and  EM  Grades,  and 
11H(B),  AV,  AO,  ATC,  AC,  and  ET  Internal  Criteria.  In  many  of  these  schools,  the  optimal 
weights  for  one  or  more  ASVAB  tests  were  close  to  zero  or  negative,  suggesting  that  the 
operational  selector  composite  could  be  improved  by  deleting  a  test. 

The  relative  incremental  validities  for  ECAT  and  ASVAB  tests  that  were  observed  for  integer 
weighted  composites  are  confirmed  in  regression-weighted  equations  in  all  but  six  instances,  that 
is.,  the  (Op+ECAT)  -  (Op+ASVAB)  validities  have  the  same  sign  as  the  Multiple  correlation 
differences  except  in  GM  and  AE  grades  and  ET,  FC,  OS,  and  AE  performance,  where  the 
differences  are  close  to  zero.  Thus,  in  most  cases,  the  results  are  not  due  to  the  weighting  method, 
but  to  the  exclusion  of  some  important  ASVAB  predictors  from  the  operational  selector 
composites.  This  exclusion  permits  the  selector  composite  to  be  improved  more  by  adding  an 
ASVAB  test  than  by  adding  an  ECAT  test. 
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Beta  Weights,  Multiple  Rs,  and  Validities  for  Composites  of  Tests 
(Operational  Composite,  Adding  One  ASVAB,  One  ECAT,  or  Assembling  Objects) 

for  School  Grade  Criteria 
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A01  (FSG)  AR  MK  El  GS  Added  Test 

OP  .146  .361  .185  .130  .670  .666 

Opt-ASVAB  .108  .321  .195  .131  .155(CS)  .683  .013  .684  .018 

OP+ECAT  .135  .358  .179  .104  .119(11)  .678  .008  -.005  .671  .005  -.013 

QP+AO _ .121  .344  .167  .115  .107(AQ)  .674  .004 _ -009 _ .670  .004 _ --014 
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Regression  Weight  Analyses  Integer  Weight  Analyses 

_ Multiple  Rs  &  Differences _ Validities  &  Differences _ 

School/Criterion/  Comp  Comp  -  Comp  Comp  - 

Composite _ Standardized  Regression  Weights _ Comp  -  Op _ (Op-fASVAB)  Comp  -  Op _ (Op+ASVAB) 
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The  integer- weighted  composite  is  AR  +  2MK  +  GS. 
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Discussion 


Although  these  conclusions  must  remain  tentative,  until  confirmed  by  cross-validation  in  new 
samples,  it  appears  that  the  operational  composites  could  be  substantially  improved  by  adding 
more  ASVAB  tests  to  most  of  them.  The  use  of  integer  weights  degrades  validity  where  exact 
weights  are  negative,  as  they  often  are  with  the  Internal  Criteria,  and  in  these  cases  the  operational 
composite  could  be  improved  by  removing  a  test.  In  many  cases  potential  validity  improvements 
from  revising  the  operational  composites  exceed  those  from  enhancing  the  ASVAB  with  the 
ECAT  tests,  although  both  improvements  would  be  desirable. 

So  far,  we  have  ignored  the  benefits  that  might  be  derived  from  improved  differential  validity 
and  classification  efficiency  that  a  larger  and  more  diverse  battery  could  produce.  However,  the 
exact  weights  derived  from  these  analyses  could  be  evaluated  to  determine  ASVAB  tests  that 
could  be  replaced  by  ECAT  tests.  Eliminating  one  of  two  ASVAB  tests  with  the  same  underlying 
dimension  for  even  a  few  selector  composites  should  improve  classification  efficiency  by  lowering 
the  intercorrelations  of  the  operational  selector  composites.  For  that  matter,  leaving  all 
operational  composites  intact  and  adding  an  ECAT  test  to  composites  of  an  occupational  group 
(where  incremental  validity  has  been  shown)  would  improve  classification  efficiency. 

Given  the  vagaries  of  the  operational  selector  composites,  the  best  index  of  the  predictive 
potential  of  ECAT  tests  remains  the  incremental  validity  determined  from  full-model  regression 
equations  in  the  earlier  sections  of  this  report.  At  the  same  time,  the  ultimate  utility  of  ECAT  can 
only  be  assessed  in  the  context  of  the  way  the  tests  will  be  used  operationally  for  selection  and 
classification  decisions. 
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General  Discussion 


We  have  analyzed  the  incremental  validities  of  ECAT  from  four  different  perspectives: 
general  ability,  ability  factors  in  multiple  regression,  tests  and  composites  in  multiple  regression, 
and  unit- weighted  selector  composites.  The  results  of  the  different  approaches  are  consistent  with 
one  another,  with  the  exception  of  the  unit-weighted  selector  composites,  where  the  problem  was 
inadequate  ASVAB  weighting.  The  incremental  validity  of  ECAT  is  not  the  result  of  tricky 
regression  weighting,  however,  because  it  shows  up  even  in  the  simple  validity  of  “g’\ 

What  are  we  to  make  of  the  fact  that  the  ECAT  validity  increments  depend  so  strongly  on 
which  criteria  are  used  to  measure  school  performance?  If  the  ECAT  project  were  to  be  evaluated 
solely  on  the  basis  of  its  ability  to  improve  prediction  of  Final  School  Grades,  or  even  school 
attrition,  it  would  have  to  be  considered  a  failure.  The  highest  significant  validity  gain  for 
predicting  FSG  was  .024,  with  many  schools  showing  no  gain  at  all.  This  result  was  expected:  we 
had  no  reason  to  think  that  psychomotor  ability,  for  example,  would  be  related  to  performance  on 
the  kinds  of  written  tests  that  form  the  basis  for  most  Final  School  Grades.  The  ASVAB  probably 
is  optimized  already  for  predicting  academic  achievement.  It  contains  tests  of  electronics,  science, 
verbal,  and  mathematical  knowledge  that  was  acquired  in  school.  Our  results  show  that  its 
corrected  validity  averages  better  than  .78  for  predicting  FSG,  an  extremely  high  value  for  most 
aptitude  batteries.  It  is,  perhaps,  surprising  that  the  ECAT  battery  could  boost  the  mean  validity 
as  high  as  it  did,  to  .80. 

On  the  other  hand,  the  incremental  validity  of  ECAT  for  predicting  hands-on  performance 
averaged  better  than  .03  and  exceeded  .10  for  some  schools.  Potentially,  these  validity  increases 
could  mean  better  hands-on  job  performance  if  recruits  were  classified  on  the  basis  of  the  relevant 
ECAT  tests.  Unfortunately,  hands-on  performance  is  seldom  measured  or  publicly  available, 
which  is  why  we  labeled  these  “internal”  criteria.  Because  hands-on  performance  is  nearly  invisible 
to  external  decision  makers  without  special  studies,  validity  improvements  are  likely  to  go 
unnoticed.  Worse,  these  criteria  are  ephemeral;  they  change  or  completely  disappear  when  the 
curriculum  changes,  as  it  frequently  does.  It  may  be  impossible  to  cross-validate  a  regression 
equation  on  the  same  school  a  year  later  because  the  criterion  no  longer  exists!  Nevertheless,  the 
same  ability  that  was  needed  to  perform  one  laboratory  exercise  may  show  up  on  a  different  one, 
or  on  subsequent  job  performance. 

Are  any  of  the  results  reproducible?  Yes,  the  ECAT  results  for  the  Army’s  11H  Heavy 
Antiarmor  Weapons  school  are  actually  cross-validations  of  earlier  studies  at  the  same  school  by 
Smith  and  Walker  (1988)  who  confirmed  a  study  by  Grafton,  Czamolewski,  and  Smith  (1989) 
showing  the  validity  of  tracking  and  spatial  tests  for  predicting  1 1H  TOW  simulator  performance. 
In  addition,  the  ECAT  study  found  that  psychomotor  and  spatial  tests  improved  prediction  of 
EVTSUM  criteria  in  two  different  samples  from  the  1 1H  school. 

Another  result  that  was  replicated  within  the  ECAT  study  itself  was  a  very  large  validity 
improvement  from  Working  Memory  and  Spatial  tests  for  predicting  Air  Traffic  Control 
operations.  The  same  result  was  found  for  two  different  samples  from  the  Air  Force  ATC  school 
and  from  the  Navy’s  AC  school.  Because  Air  Traffic  control  is  so  critical  to  human  lives  and  to 
the  safety  of  equipment,  anything  that  could  improve  the  selection  of  Air  Traffic  Controllers 
would  be  very  valuable  to  both  military  and  civilian  aviation. 
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The  ultimate  use  of  these  findings  depends  on  practical  and  economic  considerations  beyond 
the  scope  of  this  scientific  study.  It  is  not  clear,  for  example,  that  testing  every  incoming  military 
enlisted  applicant  with  the  ECAT  tests  is  an  efficient  way  to  proceed.  It  may  be  possible  to  give 
ECAT  tests  to  only  those  applicants  who  are  likely  to  be  assigned  to  1 1H,  Air  Traffic  Control,  or 
certain  other  specialties.  Although  computerized  testing  will  become  nearly  universal  with  the 
full-scale  implementation  of  CAT-ASVAB,  the  response  pedestals  needed  for  the  psychomotor 
tests  will  not  be  part  of  that  system.  Each  response  pedestal  costs  more  than  a  computer.  On  the 
other  hand,  further  research  might  develop  a  mouse-based  tracking  test  that  is  equally  effective  in 
measuring  psychomotor  ability.  In  that  case,  routine  psychomotor  testing  of  all  applicants  might 
become  feasible. 


Conclusions 

1.  Many  ECAT  tests  have  substantial  simple  validities  for  predicting  school  performance. 

2.  In  some  military  training  courses,  the  ASVAB’s  prediction  of  school  practical  performance  can 
be  substantially  improved  by  using  ECAT  tests  in  optimally-  weighted  composites. 

3.  All  ECAT  tests  have  statistically  significant  incremental  validity  over  the  ASVAB  alone. 

4.  Validity  increases  are  greatest  (averaging  5.7%)  when  laboratory  or  simulator  performance 
criteria  are  used,  rather  than  school  grades  (averaging  1.7%). 

5.  Increases  for  some  schools  are  much  larger  than  this,  while  other  schools  have  no  significant 
validity  improvement. 

6.  Very  large  cross- validated  incremental  validities  were  found  for  predicting  Air  Traffic  Control 
operations,  using  Working  memory  and  Spatial  tests. 

7.  Very  large  cross-validated  incremental  validities  were  found  for  predicting  11H  Heavy 
Antiarmor  Weapons  performance. 

8.  Factor  scores  are  more  than  98%  as  valid  as  individual  tests  in  multiple  regression,  but  relying 
on  “g”  alone  reduces  validity  by  as  much  as  8.9%  on  the  average. 

9.  ECAT  tests  can  be  used  to  broaden  the  estimate  of  general  mental  ability,  or  “g”  produced  by 
the  ASVAB.  This  enhanced  “g”  has  validity  increments  for  predicting  practical  performance 
criteria  which  are  nearly  as  large  as  the  validity  increments  from  using  all  tests  in  multiple 
regression. 

10.  The  Spatial  ability  dimension  is  essential  for  prediction.  Either  Assembling  Objects  or 
Integrating  Details  tests  can  be  omitted  from  the  battery,  provided  the  other  remains. 

11.  Both  working  memory  tests,  Mental  Counters  and  Sequential  Memory,  are  essential  for 
maximal  prediction  and  neither  can  be  deleted  from  the  ECAT  battery  without  a  significant 
decrease  in  validity. 
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12.  The  Psychomotor  tests  account  for  25%  of  the  mean  validity  gain  in  predicting  school  grades 
and  44%  of  the  gain  in  predicting  the  Internal  school  criteria,  even  after  all  other  predictors  are 
entered. 

13.  Two  P&P  tests,  Assembling  Objects  and  Figural  Reasoning,  together  can  produce  44%  and 
35%  of  the  ECAT  mean  validity  gain  for  predicting  Grades  and  Internal  Criteria,  respectively,  if 
they  are  entered  first  into  regression. 

14.  The  Spatial  Orientation  test  is  redundant  and  can  be  eliminated  from  the  ECAT  battery 
without  a  significant  decrease  in  validity,  provided  that  the  other  ASVAB  and  ECAT  tests  remain. 

15.  About  75%  of  the  incremental  validity  of  ECAT  can  be  attained  by  using  just  three  tests,  each 
one  representing  a  different  ECAT  factor. 

16.  The  best  three  ECAT  tests  for  increasing  validity  are  Two-Hand  Tracking,  Mental  Counters, 
and  Assembling  Objects. 

17.  Existing  selector  composites  can  be  improved  by  adding  ASVAB  tests  to  them.  In  most  cases, 
the  validity  improvements  from  doing  so  exceed  those  from  adding  an  ECAT  test  with  unit 
weights. 

18.  The  estimates  of  ECAT’s  incremental  validity  are  very  sensitive  to  the  type  and  quality  of 
criteria  used  to  evaluate  the  tests.  Continued  improvement  of  the  ASVAB’s  predictive  validity 
requires  improved  quality  in  the  criterion  measures  used  for  validation. 


Recommendations 

1.  Consideration  should  be  given  toward  the  eventual  incorporation  into  ASVAB  of  a  Spatial 
Ability  measure,  such  as  Assembling  Objects. 

2.  If  CAT-ASVAB  is  universally  implemented,  then  consideration  should  be  given  toward 
including  computerized  tests  of  working  memory,  such  as  Mental  Counters. 

3.  The  Mental  Counters  test  should  be  considered  for  supplementary  administration  to  potential 
students  in  the  Air  Force  and  Navy  Air  Traffic  Control  schools. 

4.  The  Two-Hand  Tracking  test  should  be  considered  for  supplementary  administration  to 
potential  students  in  the  Army  Heavy  Antiarmor  Weapons  school  (11H).  Its  cost/benefits  for 
wider  operational  testing  should  be  evaluated  under  different  concepts  of  operations. 

5.  A  variety  of  alternative  tracking  tests  should  be  investigated,  to  determine  if  a  mouse, 
trackball,  or  other  off-the-shelf  equipment  could  serve  as  well  as  slide  potentiometers  and 
joysticks.  Human  factors  work  on  alternative  tracking  item  types  and  screen  displays  should  be 
supported. 

6.  Development  of  alternate  forms  and/or  adaptive  item  pools  should  be  started  for  ECAT  tests. 
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7.  The  most  promising  EC  AT  tests  should  be  normed. 

8.  Research  on  optimal  non-negative  weighting  of  ASVAB  tests  for  maximal  cross-validated 
classification  efficiency  should  be  given  high  priority.  Operational  selector  composites  eventually 
should  be  replaced  by  these  optimal  weighting  methods. 

9.  Military  training  schools  should  be  encouraged  to  incorporate  continuously-scored  practical 
performance  measures  in  their  intermediate  and  final  grades.  The  statistical  properties  of  FSG, 
including  reliability  and  validity,  should  be  continuously  monitored  and  updated,  particularly 
following  any  shift  in  curricula. 
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Appendix  A 

ASVAB  Population  Statistics 


A-0 


Table  A-l 


Means,  Standard  Deviations,  and  Correlations  of  ASYAB  Tests 
for  Fiscal  Year  1991  Military  Applicants  (N  =  650,278) 


Variable 

AFQT 

GS 

AR 

WK 

PC 

NO 

Mean 

52.59779 

50.61501 

50.66362 

51.31136 

51.15582 

52.51222 

Std.  Dev. 

23.6111 

8.772641 

8.645407 

7.354141 

7.964013 

8.013076 

AFQT 

1 

0.746484 

0.874185 

0.829683 

0.760957 

0.479759 

GS 

0.746484 

1 

0.611079 

0.720105 

0.607873 

0.275094 

AR 

0.874185 

0.611079 

1 
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0.574273 

0.470311 

WK 

0.829683 

0.720105 

0.596262 

1 

0.731601 

0.324392 

PC 

0.760957 

0.607873 

0.574273 

0.731601 

1 

0.395913 

NO 

0.479759 

0.275094 

0.470311 

0.324392 

0.395913 

1 

cs 
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0.248734 

0.39535 

0.327805 

0.385869 

0.640106 

AS 

0.406062 

0.520184 

0.400411 

0.43662 

0.339055 

0.046951 

MK 

0.829985 

0.554223 

0.706944 

0.49678 

0.499731 

0.496123 

MC 

0.641152 

0.637695 

0.613441 

0.547276 

0.485226 

0.227886 

El 

0.547772 

0.624531 

0.486796 

0.534385 

0.444472 

0.14525 

VE 

0.860154 

0.727385 

0.625712 

0.972128 

0.86498 

0.368634 

Variable 

CS 

AS 

MK 

MC 

El 

VE 

Mean 

52.2662 

51.40873 

51.2103 

51.94077 

50.33257 

51.33477 

Std.  Dev. 

7.811829 

9.167697 

8.689045 
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7.306167 
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0.14525 

0.368634 

CS 

1 
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0.368472 

AS 
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El 
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0.369617 

0.630407 

1 

0.536475 

VE 

0.368472 

0.429827 

0.527314 

0.560015 

0.536475 
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Appendix  B 

Means,  Standard  Deviations,  and  Correlations 
of  ASVAB  and  EC  AT  Predictors 
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Mental  Counters  Proportion  Correct  SM  =  Sequential  Memory  Proportion  Correct  ID  =  Integrating  Details  Proportion  Correct 

Assembling  Objects  Proportion  Correct  T1  =  1-Hand  Tracking  Mean  1000*log(l  +  RMS(Attempted))  T2  =  2-Hand  Tracking  Mean  1000*log(l  +  RMS(Attempted)) 

Figural  Reasoning  Proportion  Correct  SO  =  Spatial  Orientation  Proportion  Correct  TI  =  Target  Identification  Mean  Clipped  Decision  RTs 


VARIABLE  T1 _ T2  FR  SO _ T1  MEMORY  SPATIAL  TRACKING  AORT  IDCT  IDDT  SORT  FRRT  AO-1  AO-2 

MEAN  2765.37  3639.16  0.6689  0.5166  1.8350  0.0053  0.0134  -0.0008  18.6950  12.5066  2.8663  13.9647  18.7153  0.6344  0.5807 

STD _ 391.7238  471.9778  0.1877  0.2473  0.6039  0.9956  1.0007  1.0032  5.5288  5.1293  0.7763  3.8645  4.6116  0.1995  0.2279 

GS  -0.1859  -0.2303  0.3441  0.3464  -0.2303  0.2204  0.3929  -0.2236  0.0196  0.0081  -0.1185  -0.0270  0.0109  0.2741  0.3084 

AR  -0.1808  -0.2111  0.4494  0.3915  -0.1573  0.4710  0.4505  -0.2106  0.0776  0.0511  -0.1002  0.0116  0.0647  0.3251  0.3178 
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Range-Corrected  Means,  Standard  Deviations,  and  Intercorrelations  of  Predictors  and  Other  Measures 
_ (N  ~  10,963) _ 

VARIABLE  GS _ AR  WK _ PC _ NO _ CS _ AS _ MK _ MC  El  VE  AFQT  CT _ SM _ ID _ AO 

MEAN  50.6150  50.6636  51.3114  51.1558  52.5122  52.2662  51.4087  51.2103  51.9408  50.3326  51.3518  51.0176  0.6772  0.6554  0.7209  0.5746 
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Mental  Counters  Proportion  Correct  SM  =  Sequential  Memory  Proportion  Correct  ID  =  Integrating  Details  Proportion  Correct 

Assembling  Objects  Proportion  Correct  T1  =  1  -Hand  Tracking  Mean  1000*log(l  +  RMS(Attempted))  T2  =  2-Hand  Tracking  Mean  1000*iog(l  +  RMS(Attempted)) 

Figural  Reasoning  Proportion  Correa  SO  =  Spatial  Orientation  Proportion  Correct  TI  =  Target  Identification  Mean  Clipped  Decision  RTs 
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AO  Proportion  Correct  of  Final  17  Items  (Jig-saw) 


Appendix  C 

Factor  Loadings  and  Scoring  Weights 
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Promax  Oblique  Factor  Pattern  Loadings  for  ASVAB  Tests 
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Promax  Oblique  Factor  Pattern  Loadings  for  the  Combined  ASVAB  and  ECAT  Battery 
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Factor  Scoring  Weights  for  ASVAB  Tests 
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Test  Scoring  and  Subject  Deletion  Rules 
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Test  Scoring  and  Subject  Deletion  Rules 


David  L.  Alderton 

For  Mental  Counters,  Integrating  Details,  Figural  Reasoning,  Assembling  Objects,  and  Spatial 
Orientation  the  final  summary  score  was  the  proportion  correct  of  the  total  number  of  items.  For 
Sequential  Memory,  the  final  score  was  the  proportion  of  digits  correct  (of  all  possible). 

Subject  deletion  rules  were  by  and  large  empirically  based.  For  Mental  Counters  and  Sequential 
Memory,  Jerry  Larson  provided  the  cut  scores.  For  Integrating  Details,  David  Alderton  provided  the  cut 
scores.  For  all  three  of  these  tests,  the  score  cut  values  were  based  on  the  ECAT  database  and  past 
empirical  research.  For  Figural  Reasoning,  Assembling  Objects,  and  Spatial  Orientation  the  cut  scores 
were  set  by  David  Alderton  since  AIR  and  ARI  said  that  there  were  no  established  cut  score  rules  or 
subject  screening  algorithms.  The  conservative  cut  scores  for  these  tests  were  based  on  logic  (subjects 
attempting  less  than  1/3  of  the  items  were  eliminated)  and  empirical  score  distributions.  Cut  scores  for  the 
three  response  pedestal  tests  were  provided  by  AIR  (see  notes). 

Mental  Counters 

MCPCOR  <  .19 

Sequential  Memory 

SMPDCOR  <  .15 
SMPIAT  <  .65 

Integrating  Details 
IDJP  >  9 
IDPAT  <  .75 
IDPCOR/IDPCAT  <  .38 

Figural  Reasoning 

SRPAT  <  .33 
SRPCOR  <  (2.5/30) 

Assembling  Objects 

AOPAT<  (11/32) 

AOPCOR  <  (2.75/32) 

Spatial  Orientation 

ORPAT  <  (8/24) 

ORPCOR  <  (2/24) 

Target  Identification  (see  attached  notes  on  scoring  response  pedestal  tests) 

Items  that  time  out  are  treated  as  valid  but  wrong. 

Items  with  Decision  Times  <  .1  are  treated  as  invalid. 

Items  with  Movement  Times  <  .01  are  treated  as  invalid. 

If  the  proportion  of  VALID  items  correct  <  1/3  the  subject  is  eliminated 

One-  and  Two-Hand  Tracking  (see  attached  notes  on  scoring  response  pedestal  tests) 

If  two  (or  more)  items  are  missing  the  subject  is  eliminated. 
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Notes  on  Response  Pedestal  Tests 

The  following  is  a  comment  from  the  SAS  file  ARI.SAS  on  the  ECAT  (193)  IBM  disk.  The 
comments  are  self-explanatory.  I  developed  the  scoring  code  based  on  conversations  with  Scott  Oppler  and 
Dianne  Steele.  After  writing  the  SAS  code,  I  scored  the  same  data  they  did  (Fort  Knox,  I  believe),  faxed 
them  the  descriptive  statistics,  and  they  notified  me  that  our  results  were  in  perfect  agreement.  I  later  gave 
this  information  (and  my  SAS  code)  to  RGI  who  used  it  to  create  the  .merge  files. 


/*« 

/* 

/* 

/* 

/* 

/* 

/*« 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/* 

/*= 

/* 

/* 

/* 

/* 

/* 

/* 

/*= 


THESE  NOTES  ARE  BASED  ON  A  4/17/92  CONVERSATION  WITH  */ 

SCOTT  OPPLER  AND  DIANNE  STEELE  FROM  AIR  ON  HOW  THE  */ 

TARGET  IDENTIFICATION  TEST  SHOULD  BE  SCORED  AND  HOW  */ 

PEOPLE  SHOULD  BE  SCREENED.  THERE  IS  ALSO  A  NOTE  ON  */ 

SCREENING  SUBJECTS  FOR  THE  TRACKING  TESTS  */ 

TARGET  IDENTIFICATION  */ 

^  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  A  *  J 

THE  TEST  SHOULD  BE  SCORED  IN  THE  FOLLOWING  STEPS:  */ 

A.  IF  AN  ITEM  TIMES  OUT,  SET  THE  MT,  DT  SET  TO  MISS  */ 

BUT  SET  AC  TO  WRONG  */ 

B.  IF  AC  IS  WRONG,  SET  MT  AND  DT  TO  MISSING  */ 

C.  IF  DT  IS  <  .1  THEN  SET  DT,  MT,  ACC  TO  MISSING  */ 

D.  IF  MT  IS  <  .01  THEN  SET  DT,  MT,  ACC  TO  MISSING  */ 

E.  IF  RECALCULATED  ACCURACY  IS  <  1/3  THEN  SET  FINAL  */ 

SCORES  TO  MISSING  */ 

*/ 

FINAL  SCORES  */ 

A  A  A  A  A  A  A  A  A  A  A  A  *  j 

PROPORTION  CORRECT  (AFTER  SCREENS)  */ 

MEDIAN  MT  ACROSS  ALL  VALID,  CORRECT  ITEMS  */ 

AVERAGE  OF  THE  CLIPPED  EASY  MEAN  DT  AND  THE  */ 

CLIPPED  HARD  MEAN  DT :  */ 

(1)  AFTER  ABOVE  SCREENS,  SORT  ITEMS  BY  EASY  AND  HARD  */ 

(2)  THROUGH  OUT  THE  MIN  AND  MAX  DTS  WITHIN  A  GROUP  */ 

(3)  CALCULATE  THE  (CLIPPED)  W/I  GROUP  MEAN  */ 

(4)  AVERAGE  THE  TWO  CLIPPED  MEANS  =  MDT  */ 

*/ 

FOLLOWING  THE  THREE  PRACTICE  ITEMS,  THE  EASY  AND  ITEM  */ 
(ORDINAL)  POSITIONS  ARE:  */ 

EASY  ITEMS  ARE:  1,  3,  5,  6,  7,  10,  13,  14,  15,  17,  18,  */ 

19,  23,  26,  27,  28,  30,  32,  34,  36  */ 

HARD  ITEMS  ARE:  2,  4,  8,  9,  11,  12,  16,  20,  21,  22,  24,  */ 

25,  29,  31,  33,  35  */ 

==========================================================*/ 

1  AND  2  HAND  TRACKING  TESTS  */ 

A  A/V  AAA  A  A  A  A  A  A  A /\  A A  AAA  A  AAA  A  A  A  *  j 

DROP  SUBJECTS  THAT  HAVE  MORE  THAN  2  MISSING  ITEMS  */ 

SCOTT  AND  DIANNE  THINK  THAT  THE  AVERAGE  OF  THE  ITEM  LEVEL*/ 
SCORES,  I.E.,  THE  AVERAGE  OF  THE  LOG [(RMS  DIST)  +1],  */ 

IS  BEING  TRUNCATED  IN  ECAT,  SO  I'LL  RECALCULATE  IT  */ 

===================================================“==='===*/ 
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Test/Retest  Results  for  the  ECAT  Battery1 


Gerald  E.  Larson 
David  L.  Alderton 


There  are  several  reasons  why  it  is  critical  to  conduct  a  test-retest  analysis  of  new  tests.  First,  the 
reliability  coefficient  can  be  used  to  disattenuate  correlations  between  predictor  and  criterion  measures, 
providing  a  better  estimate  of  the  true  relationship.  Second,  reliability  is  often  a  consideration  in 
decisions  regarding  test  implementation.  Third,  retest  studies  provide  information  on  practice  effects 
and  perhaps  even  on  coachability.  For  these  reasons  the  ECAT  battery  was  administered  twice  across 
4-  to  5- week  intervals.  The  details  of  the  study  are  presented  below. 


Method 


Subjects 

While  it  would  have  been  optimal  to  test  military  recruits,  military  scheduling  considerations  made 
recruit  testing  impractical  for  the  current  research.  Thus,  high  school  and  junior  college  students  in  the 
San  Diego  vicinity  were  recruited  as  subjects,  with  the  restrictions  that  subjects  must  be  between  the 
ages  of  16  and  26,  with  the  total  sample  having  no  more  than  35%  females  and  no  less  than  60% 
Caucasians.  The  purpose  of  these  restrictions  was  to  ensure  comparability  between  the  sample  and 
military  recruits.  As  an  incentive  to  participate  in  the  study,  each  subject  was  paid  $15.00  for  each  test 
session  plus  a  $40.00  bonus  upon  completing  the  retest  session  and  submitting  a  copy  of  the  subjects' 
high  school  transcripts.  Further  details  of  the  data  collection  are  provided  in  Brantner  (1992). 


Three  hundred  and  thirteen  subjects  (223  males,  90  females)  completed  both  test  sessions.  They 
averaged  19.3  years-of-age,  with  a  standard  deviation  of  2.8.  The  ethnic  breakdown  was  as  follows: 
73%  Caucasian,  10%  Hispanic,  6%  Asian,  4%  Filipino,  3%  African-American,  4%  "Other." 


1  Presented  at  the  Centennial  Convention  of  the  American  Psychological  Association;  Washington,  D.  C.,  August  1992 
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Aptitude  Tests 


Each  subject  completed  an  approximately  3-hour  battery  of  10  computerized  tests,  presented  on 
Hewlett-Packard  Integral  microcomputers  operating  under  UNIX™.  Nine  of  the  10  tests  comprised 
the  actual  ECAT  battery.  The  tenth,  “Perceptual  Speed,”  was  included  as  a  supplemental  measure 
about  which  information  was  desired.  All  tests  were  written  in  standard  C.  Tests  1-7  below  used  a 
simplified  keyboard.  The  keyboard  was  modified  by  using  a  plastic  mask  that  revealed  only  the 
designated  response  keys  along  with  a  key  labeled  HELP  that  could  be  pressed  during  testing  to 
suspend  the  program  and  request  assistance.  The  S,  F,  H,  K,  and ;  keys  were  relabeled  as:  A,  B,  C,  D, 
and  E.  The  space  bar  was  relabeled  ENTER.  The  numeric  keypad  keys  retained  their  meanings.  Tests 
8-10  below  (Target  Identification,  One-hand  Tracking,  and  Two-hand  Tracking)  used  a  custom  built 
“response  pedestal”  with  response  buttons,  sliders,  and  a  joy  stick. 


1 .  Integrating  Details  -  A  complex  40  item  spatial  problem  solving  test.  Each  item  consists  of  two 
separate  screens.  The  first  screen  contains  from  2  to  6  regular  geometric  puzzle  pieces  that  must  be 
mentally  brought  together  to  form  a  completed  object.  This  is  much  like  a  jig-saw  puzzle.  Having 
connected  all  of  the  puzzle  pieces,  the  individual  must  remember  the  final  object,  then  press  a  response 
key  indicating  that  she/he  is  ready.  Once  the  key  is  pressed,  the  puzzle  pieces  are  replaced  by  a  new 
screen  with  a  single  completed  object.  The  subject  must  indicate  if  the  completed  object  shown  is  a 
product  of  the  original  puzzle  pieces.  There  are  three  dependent  measures  for  each  trial;  time  spent 
studying  the  puzzle  pieces,  time  spent  deciding  if  the  completed  form  is  valid,  and  response  accuracy. 


2.  Mental  Counters  -  Mental  Counters  is  a  complex  40  item  working  memory  test.  Each  screen 
contains  three  horizontal  lines,  arrayed  left  to  right.  Each  line  represents  a  counter  with  an  initial  value 
of  zero.  During  an  item,  boxes  appear  sequentially,  one  at  a  time,  either  above  or  below  one  of  the 
three  lines.  If  a  box  appears  above  a  line,  the  value  for  that  counter  is  incremented  by  +1.  If  a  box 
appears  below  a  line,  that  counter  is  decremented  by  - 1 .  On  each  trial  either  5  or  7  boxes  appear.  The 
boxes  appear  at  one  of  two  rates,  either  one  every  1.33  seconds  or  one  every  .75  seconds.  The 
subject's  task  is  to  make  a  series  of  rapid  calculations  and  to  select,  from  a  four-alternative  multiple 
choice  menu,  the  set  of  correct  final  counter  values.  Number  of  correct  responses  is  used  as  the 
summaiy  score. 

3.  Sequential  Memory  -  Sequential  Memory  is  another  complex  test  of  working  memoiy.  Each 
item  consists  of  three  to  five  horizontally  arrayed  dots  on  the  screen.  Each  dot  is  given  a  numerical 
value;  these  must  be  memorized.  The  item  is  then  presented  in  a  series  of  5  to  7  "calls"  to  the  dots; 
where  each  call  is  announced  by  briefly  turning  one  of  the  dots  into  an  "X."  The  person  must  report 
the  digit  string  that  corresponds  to  the  order  that  the  dots  were  "called."  In  the  second  half  of  the  test, 
after  all  the  calls  for  an  item  have  been  made,  the  examinee  is  told  to  translate  each  number  in  the 
ordered  number  list  into  a  different  number  and  then  type  in  the  new  ordered  list.  There  are  10  items  in 
the  first  half  of  the  test  and  25  in  the  second  half  of  the  test.  The  dependent  variable  is  the  proportion 
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of  digits  correctly  reported  by  the  examinee. 


4.  Spatial  Reasoning  -  A  figural  inductive  reasoning  (or  series  extrapolation)  test,  similar  to  the 
Cognitive  Abilities  Figural  subtests.  Items  use  a  combination  of  geometric  forms  and  arbitrary  figures 
presented  in  a  series  of  four  frames.  The  subject's  task  is  to  induce  the  transformation  rule  controlling 
the  series  and  then  select  one  of  five  alternatives  that  correctly  completes  the  series.  The  dependent 
variable  is  number  correct  across  the  30  items.  There  is  a  12  minute  time  limit. 


5.  Perceptual  Speed  -  Perceptual  Speed  (Alderton,  1990)  is  a  clerical/perceptual  speed  test.  Each 
item  consists  of  two  side-by-side  symbol  strings  of  the  same  length.  The  examinee's  task  is  to 
determine  whether  the  two  symbol  strings  are  identical,  and  to  make  these  judgements  as  rapidly  as 
possible  while  maintaining  90%  accuracy.  Symbol  string  length  is  systematically  varied  from  1  to  7 
elements.  The  test  is  divided  into  3  subtests  based  on  string  content:  Numbers  (56  items),  letters  (56 
items),  or  abstract  stick  figures  (60  items).  Each  item  type  (number  of  elements  X  symbol  type)  has  a 
minimum  and  maximum  response  time  bracket  associated  with  it.  If  an  examinee  responds  too  quickly 
or  too  slowly  she/he  is  warned  to  slow  down  or  speed  up.  Cumulative  accuracy  is  retained  and  used  in 
feedback  after  every  10-14  items.  To  control  for  speed/accuracy  tradeoffs,  the  examinee  is  warned  to 
slow  down  if  accuracy  drops  below  85%  or  to  speed  up  if  accuracy  goes  above  95%.  The  primary 
dependent  variable  is  the  average  rate  score  across  the  three  subtests  where  rate  is  defined  as  the 
proportion  correct  divided  by  the  geometric  mean  of  item  reaction  times. 


6.  Assembling  Objects  -  A  spatial  construction  test.  Each  item  consists  of  a  frame  with  several 
(2-6)  separate  elements.  The  subject's  task  is  to  choose,  from  four  alternatives,  the  answer  that 
correctly  represents  how  the  elements  should  be  connected.  There  are  32  items  in  the  test.  The  first  15 
items  are  semi-mechanical  items  with  labels  indicating  how  the  elements  should  be  connected.  The 
final  17  items  in  the  test  consist  of  small  jigsaw  puzzles  similar  to  those  used  in  the  Minnesota  Paper 
Form  Board  test.  There  are  no  labels  showing  how  the  puzzle  pieces  are  to  be  connected  but  only  one 
of  the  four  answer  choices  includes  all  of  the  puzzle  elements.  The  dependent  variable  is  the  number  of 
correct  items  solved  in  16  minutes. 


7.  Spatial  Orientation  -  A  spatial  perspective  test.  Each  item  consists  of  an  environmental  view, 
such  as  a  bridge  over  a  river  or  a  farm  house.  In  each  view  the  horizon  is  apparent.  These  views  are 
rotated  away  from  the  "natural"  horizon  in  a  frame.  At  the  bottom  of  the  frame  is  a  circle  with  a  dot  on 
the  perimeter.  The  subject’s  task  is  to  rotate  the  frame  around  the  view  until  it  corresponds  with  the 
natural  horizon  of  the  view  and  determine  where  the  dot  on  the  circle  would  be  located.  This 
information  is  then  used  to  select  which  of  5  alternatives  correctly  shows  where  the  dot  would  be  on 
the  circle  (following  the  rotation).  The  dependent  variable  is  the  number  of  items  (of  24)  solved 
correctly  in  the  allotted  time. 
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The  next  3  tests  use  the  ECAT  response  pedestal  to  input  responses. 


8.  Target  Identification  -  A  hybrid  test  combining  aspects  of  choice  reaction  time  and  spatial 
mental  rotation  tests.  Each  item  consists  of  a  figure  in  the  top  half  of  the  screen  and  three  alternative 
figures  in  the  bottom  half  of  the  screen.  The  correct  answer  is  the  alternative  (at  screen  bottom)  that 
represents  the  same  object  as  the  standard,  even  though  the  standard  may  be  distorted  (e.g.,  rotated, 
shrunken,  or  both)  relative  to  the  answer  choice.  (Answer  choices  are  always  presented  in  a  "natural" 
upright  position)  The  examinee’s  task  is  to  select  the  correct  alternative  as  rapidly  as  possible.  The 
figures  are  schematic  line  drawings  of  simple  objects,  such  as  trucks,  helicopters,  and  tanks.  Before 
each  item  the  subject  is  required  to  hold  down  4  "home"  buttons,  two  on  the  left  and  two  on  the  right. 
The  "home"  buttons  are  located  on  the  sides  and  top  of  the  response  pedestal  in  such  a  way  that  one 
must  use  thumbs  and  forefingers  to  hold  the  buttons  down,  thus  freezing  the  hands  in  place.  While  all 
four  buttons  are  simultaneously  depressed  the  item  is  presented.  As  soon  as  the  examinee  decides 
upon  an  answer,  either  hand  may  be  used  to  press  the  button  (on  the  top  of  the  pedestal)  that 
corresponds  to  the  selected  alternative.  As  soon  as  any  of  the  four  "home"  buttons  are  released  the 
alternatives  are  masked  (blacked  out).  The  dependent  variable  is  the  average  correct  decision  time 
where  decision  time  is  defined  as  the  time  between  item  presentation  and  "home"  button  release.  There 
are  36  items  administered  with  a  maximum  7  minute  total  test  time. 


9.  One-Hand  Tracking  -  A  psychomotor  test  that  uses  a  response  pedestal  Each  item  begins  with 
a  "path"  on  the  computer  screen.  The  path  is  simply  a  contiguous  string  of  lighted  screen  pixels.  The 
path  goes  up/down  and/or  right/left,  parallel  with  the  sides  of  the  screen  and  makes  only  90  degree 
turns.  At  one  end  of  the  path  is  a  diamond  indicating  the  path's  termination  point.  Starting  at  the  other 
end  is  a  box  that  travels  forward  along  the  path.  The  subject  moves  a  joy-stick  that  controls  the 
movement  of  a  "cross-hair."  The  subject's  task  is  to  keep  the  cross-hair  on  the  moving  box.  Items  vary 
in  terms  of  the  length  of  the  path  which  is  inversely  related  to  the  speed  at  which  the  box  moves  (total 
item  duration  is  thus  constant).  For  each  item,  the  "score"  is  the  average  absolute  Cartesian  pixel 
distance  between  the  cross-hair  and  the  moving  box  (a  distance  reading  is  taken  every  50  msec  during 
the  item).  There  are  18  items.  The  dependent  variable  for  the  test  is  the  average  of  the  18  item  scores. 


10.  Two-Hand  Tracking  -  Another  psychomotor  test  that  has  exactly  the  same  structure  and  task 
constraints  as  One-Hand  Tracking  described  above.  The  only  difference  is  that  movement  of  the  cross¬ 
hair  is  controlled  by  two  slide  potentiometers.  One  of  the  slides  controls  the  horizontal  (left/right) 
movement  of  the  cross-hair  while  the  second  slide  controls  the  vertical  (up/down)  motion  of  the  cross 
hair.  One  hand  must  be  used  for  each  slide  control.  The  slides  are  arranged  such  that  the  horizontal 
slide's  physical  movement  is  right  and  left  while  the  vertical  slide's  physical  movement  is  up  and  down. 
Number  of  items,  test  scoring,  and  final  test  score  are  the  same  as  above. 
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Two  test  administration  sequences  were  used,  corresponding  to  odd  and  even  social  security 
numbers. 


Even  SSN_  Seguence 
Sequential  Memory 
Spatial  Reasoning 
Integrating  Details 
One  Hand  Tracking 
Target  Identification 
Two  Hand  Tracking 
Assembling  Objects 
Spatial  Orientation 
Mental  Counters 
Perceptual  Speed 


Odd  SSN  Seguence 
Integrating  Details 
Spatial  Reasoning 
Sequential  Memory 
Two  Hand  Tracking 
Target  Identification 
One  Hand  Tracking 
Mental  Counters 
Spatial  Orientation 
Assembling  Objects 
Perceptual  Speed 


Perceptual  Speed  was  always  administered  last  because  it  was  not  part  of  the  ECAT  battery  per 
se. 


Results 


Prior  to  the  main  analyses  the  data  were  trimmed  to  eliminate  subjects  who  scored  10%  or  more 
below  chance  on  the  power  tests.  Also,  subjects  were  eliminated  if  their  scores  declined  50%  or  more 
from  session  one  to  session  two,  or  if  the  score  for  either  session  lagged  four  standard  deviations  below 
the  sample  mean.  Finally,  speed  test  scores  were  discarded  if  accuracy  was  below  70%.  These  data 
editing  rules  were  designed  to  eliminate  unmotivated  or  severely  impaired  examinees.  Upon 
implementing  these  rules,  the  proportion  of  subjects  excluded  from  the  analyses  ranged  from  a  high  of 
6%  on  Assembling  Objects  and  Mental  Counters  to  a  low  of  .3%  on  One-hand  Tracking. 


Practice  Effects 

Descriptive  statistics  and  practice  effects  for  the  remaining  subjects  are  shown  in  Table  E-l.  As 
can  be  seen,  practice  effects  (reflecting  improved  performance)  were  significant  for  all  tests  except 
Assembling  Objects.  Given  the  relative  novelty  of  the  experimental  measures,  some  improvement  with 
practice  was  to  be  expected.  In  many  cases,  however,  improvements  were  of  little  practical  importance 
despite  statistical  significance.  For  example,  note  the  slight  (less  than  one  tenth  of  a  standard  deviation) 
though  significant  gain  for  the  Integrating  Details  test.  In  general,  score  gains  were  greatest  for 
speeded  and/or  psychomotor  tests  (especially  Two-hand  Tracking)  and  it  is  therefore  this  category  of 
measures  which  should  be  the  focus  of  concern  for  issues  such  as  practice  and  coaching. 
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Table  E-l 


Descriptive  Statistics  and  Practice  Effects 


Session  1 

Session  2 

Session  Difference 

t 

2-tail 

Variable 

Meanl 

SD1 

Mean2 

SD2 

Value 

df 

Prob. 

PSRATE 

.709 

.089 

.753 

.101 

-12.89 

308 

.000 

SEQMEM 

.707 

.140 

.761 

.141 

-10.76 

307 

.000 

SP_REAS 

.692 

.199 

.733 

.175 

-5.22 

295 

.000 

INTEGRATE 

.773 

.132 

.784 

.128 

-2.24 

306 

.026 

ASSEMBLE 

.673 

.214 

.686 

.211 

-1.77 

292 

.079 

ORIENT 

.530 

.258 

.628 

.256 

-9.21 

294 

.000 

COUNTERS 

.781 

.160 

.795 

.183 

-2.04 

292 

.042 

TARGETTD 

1.66 

.568 

1.37 

.504 

14.99 

310 

.000 

TRACK1 

2913 

432 

2777 

475 

9.35 

312 

.000 

TRACK2 

3863 

531 

3549 

619 

21.03 

309 

.000 

Reliabilities 

Test  reliabilities  are  shown  in  Table  E-2.  Retest  reliabilities  range  from  .75  to  .91,  with  a  median 
of  .81.  These  figures  compare  favorably  with  ASVAB  retest  reliabilities,  which  range  from  .63  to  .88, 
with  a  median  of  .79  (Wolfe,  in  preparation).  Internal  consistency  estimates  are  also  quite  acceptable, 
ranging  from  .78  to  .97  across  both  sessions.  In  general,  reliabilities  were  somewhat  higher  for 
speeded  and/or  psychomotor  tests  that  for  power  tests.  Since  as  noted  above  the  former  also  showed 
the  greatest  practice  effects,  one  may  infer  that  practice  caused  an  upward  shift  in  the  psychomotor 
score  distribution  without  a  substantial  reordering  of  individual  ranks. 
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Table  E-2 


Test  Reliabilities 


Variable 

Session  1 

Alpha 

Session  2 

Alpha 

Retest 

Reliability 

PS  RATE 

.95a 

,94a 

.86 

SEQMEM 

.88 

.89 

.81 

SP_REAS 

.87 

.86 

.75 

INTEGRATE 

.79 

.78 

.79 

ASSEMBLE 

.87 

.89 

.83 

ORIENT 

.89 

.90 

.75 

COUNTERS 

.89 

.91 

.79 

TARGETED 

.97a 

.97* 

.80 

TRACK1 

.97a 

.97a 

.84 

TRACK2 

.97a 

.97a 

.91 

a  Split-half  reliabilities. 


Gender  Effects 

Table  E-3  shows  test  performance  as  a  function  of  gender.  Females  scored  significantly  below 
males  on  five  of  the  ten  tests;  two  of  these  tests  were  spatial  in  nature  (Integrating  Details  and  Spatial 
Orientation)  and  three  were  psychomotor  (Target  ID,  One-  and  Two-hand  Tracking).  To  provide  a 
better  context  for  these  findings,  it  should  be  noted  that  there  were  no  gender  differences  in  academic 
standing  (Le.,  grade  point  average)  within  the  sample,  nor  were  there  differences  on  the  ECAT 
reasoning  and  working  memory  tests.  Therefore,  there  is  no  reason  to  believe  that  the  gender  effects 
reflect  underlying  general  intelligence  differences  rather  than  specific  spatial  and  psychomotor 
differences. 
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Gender  Differences  in  Test  Performance 
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Further  examination  of  Table  E-3  reveals  significant  Gender  by  Session  interactions  for  the 
Sequential  Memory  and  Integrating  Details  tests.  In  both  cases  the  significant  interaction  reflects  the 
fact  that,  given  practice,  females  improved  substantially  more  than  males  on  these  tests.  For 
Integrating  Details,  this  female  score  gain  served  to  diminish  an  initial  (ie,  session  1)  male  advantage. 
For  Sequential  Memory,  females  advanced  ahead  of  males  in  session  2  after  equivalent  performance  in 
session  1. 


Correlations 


Table  E-4  shows  correlations  of  the  tests  with  high  school  grade  point  average  (GPA)  and  with 
each  other.  For  this  analysis  Session  1  and  Session  2  scores  were  averaged  to  create  one  global  score 
per  test.  All  variables  except  Target  ID  were  significantly  correlated  with  GPA.  Prediction  of  GPA 
would  seem  to  be  strongest  when  based  on  working  memory  scores  (ie..  Mental  Counters  and 
Sequential  Memory)  and  mixed  when  based  on  spatial  scores,  with  Integrating  Details  faring  the  best  in 
the  latter  category.  The  table  also  shows  numerous  significant  correlations  among  the  experimental 
tests  themselves.  The  strength  and  pattern  of  these  test  interrelationships  is  simplified  via  the  factor 
pattern  shown  in  Table  E-5,  which  shows  that  the  global  scores  for  the  nine  ECAT  tests  (ignoring,  for 
the  moment,  the  non-ECAT  perceptual  speed  test)  cluster  into  two  dimensions,  the  first  representing 
cognitive  problem  solving  abilities  and  the  second  representing  psychomotor  skills.  When  the  data 
from  sessions  one  and  two  are  analyzed  separately  the  same  two-factor  pattern  emerges,  suggesting 
that  the  pattern  shown  in  Table  E-5  is  a  fairly  reliable  portrayal  of  the  dimensionality  of  the  ECAT 
battery,  and  that  limited  practice  does  not  overtly  change  the  pattern. 


Table  E-4 

Pearson  Correlation  Coefficients 


Variable 

GPA 

(!) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

1. 

PSRATE 

.20** 

1.00 

2. 

SEQMEM 

.36** 

.32** 

1.00 

3. 

SP_REAS 

.30** 

.24** 

.60** 

1.00 

4. 

INTEGRATE 

.35** 

19** 

.58** 

.65** 

1.00 

5. 

ASSEMBLE 

.26** 

.27** 

.62** 

.68** 

.72** 

1.00 

6. 

ORIENT 

.27** 

.14** 

.55** 

.62** 

.67** 

.67** 

1.00 

7. 

COUNTERS 

.38** 

.35** 

.70** 

.63** 

.64** 

.68** 

.55** 

1.00 

8. 

TARGETID 

-.07 

-.28** 

-.23** 

-.24** 

-.27** 

-.33** 

-.28** 

-.26** 

LOO 

9. 

TRACK1 

-.16** 

-.23** 

-  33** 

-.45** 

-.46** 

-.43** 

-.43** 

-.42** 

.37** 

1.00 

10. 

TRACK2 

-.11* 

-.21** 

-.33** 

-.41** 

-.48** 

-.46** 

-.47** 

-.41** 

44** 

.84** 

*  p  <  .05 


**  pc.Ol 
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Table  E-5 


Factor  Analysis  of  ECAT  Tests 


Variable 

g  Factor 

— 

Varimax  Rotated 

Problem 

Solving  Psychomotor 

SEQMEM 

.72 

1 

l 

1 

.79 

-.09 

SPJREAS 

.79 

1 

| 

.80 

-.21 

INTEGRATE 

.82 

1 

1 

.78 

-.29 

ASSEMBLE 

.85 

1 

1 

1 

.82 

-.29 

ORIENT 

.78 

1 

1 

1 

.75 

-.27 

COUNTERS 

.79 

1 

1 

1 

.81 

-.19 

TARGETED 

-.47 

1 

1 

I 

-.14 

.66 

TRACK1 

-.67 

1 

1 

t 

-.24 

.88 

TRACK2 

-.69 

I 

» 

_ I _ 

-.27 

.89 

54%  of  Variance  accounted  for  by  g. 


Discussion  And  Conclusions 


Results  from  our  test/retest  administration  of  the  ECAT  battery  are,  for  the  most  part,  highly 
encouraging.  Test  reliabilities  are  at  least  as  good  as  those  for  the  operational  ASVAB,  and  although 
the  present  study  was  not  specifically  designed  as  a  validation  effort  the  correlations  of  ECAT  tests 
with  high  school  GPA  were  numerous  and  highly  significant.  The  latter  result  supports  an  optimistic 
prognosis  for  test  validities  in  military  education  and  training  settings. 


Of  concern  in  the  present  study  are  the  significant  practice  effects  observed  for  nearly  all  tests,  and 
the  female  score  deficit  observed  on  some  spatial  tests  and  all  psychomotor  tests.  These  issues  should 
be  addressed  by  follow-on  research  prior  to  operational  use  of  ECAT  tests.  An  important  question  is 
whether  adding  more  practice  items  at  the  beginning  of  the  tests  can  stabilize  performance  prior  to  the 
administration  of  operational  items.  With  regard  to  gender  differences,  follow-on  analyses  must 
include  actual  military  criterion  performance  measures.  For  example,  if  females  under-perform  (relative 
to  males)  on  criterion  as  well  as  predictor  measures  then  the  latter  deficit  does  not  reflect  test  bias.  A 
finding  of  gender  equivalence  on  the  criteria  would,  however,  suggest  that  the  test  is  a  biased  predictor 
and  that  alternative  tests  or  administration  formats  must  be  sought. 
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Appendix  F 

Group  Differences  in  the  EC  AT  Validity  Study 
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Group  Differences  in  the  ECAT  Validity  Study 


David  L.  Alderton 


Subjects  were  divided  into  six  ethnic  groups:  White,  Black,  Asian,  Hispanic,  North 
American  Indian,  and  Other.  The  categories  are  a  combination  of  the  population  and  ethnic 
group  codes  taken  from  enlistment  records.  White  was  defined  by  the  population  code 
Caucasian  (C)  and  the  ethnic  code  none  (Y).  Black  was  defined  by  the  population  code 
Negroid/African/Black  (N)  unless  a  Hispanic  ethnic  code  was  also  checked  (then  the  person 
would  be  defined  as  Hispanic,  see  below).  The  Asian  group  was  defined  by  the  population  code 
Asian/Mongoloid/Yellow  (M)  and/or  ethnic  codes  for  other  Asian  descent  (3),  Filipino  (5), 
Chinese  (G),  Japanese  (J),  Korean  (K),  Vietnamese  (V),  Melanesian  (E),  Micronesian  (W), 
Polynesian  (L),  and  other  Pacific  Island  descent  (Q).  Regardless  of  the  population  code,  the 
Hispanic  group  was  defined  by  ethnic  codes  for  other  Hispanic  descent  (1),  Puerto  Rican  (4), 
Mexican  (6),  Cuban  (9),  and  Latin  American  with  Hispanic  descent  (S).  The  North  American 
Indian  group  was  defined  by  the  population  code  for  American  Indian/Red  (R)  and  by  ethnic 
codes  for  U.S./Canadian  Indian  Tribes  (2),  Eskimo  (7),  and  Aleut  (8).  A  final  group  labeled 
Other  was  created  from  the  population  code  Other  (X)  and  the  ethnic  codes  Other  (X)  (unless 
Caucasian),  Indian  (from  India;  D),  and  Unknown  (Z).  The  distribution  of  subjects  across  the  six 
ethnic  groups  was: 


Group 

N 

Percent 

White 

7636 

71 .1 

Black 

1771 

16.5 

Asian 

241 

2.2 

Hispanic 

631 

5.9 

No.  Am.  Indian 

85 

0.8 

Other/Unknown 

369 

3.4 

Total 

10733 

100 

For  the  analyses  that  follow,  the  North  American  Indian  group  was  eliminated  because  the 
sample  size  was  too  small  to  be  meaningful.  The  Other  category  was  also  eliminated  since  the 
results  could  not  be  interpreted. 

Entries  in  the  following  table  were  computed  in  several  steps.  First,  for  each  of  the  10 
ASVAB  tests  and  9  ECAT  tests,  z-scores  were  computed  by  taking  the  first  named  ethnic 
group's  mean  minus  the  second  named  ethnic  group's  mean,  divided  by  the  standard  deviation 
from  the  total  sample.  Second,  the  within  test  battery  median,  mean,  minimum,  maximum,  and 
range  of  z-scores  were  computed:  these  are  the  entries  in  the  Table.  For  example,  the  first  entry 
(.727)  is  the  median  z- score  advantage  of  Whites  (since  it  is  positive)  over  Blacks  across  the  ten 
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ASVAB  tests.  For  the  ECAT  tests,  the  median  advantage  of  Whites  over  Blacks  is  .656  z-score 
units.  (Note  that  for  the  ECAT  tests,  the  three  response  pedestal  tests  [Target  Identification, 
One-  and  Two-Hand  Tracking]  use  speed  and  error  scores  so  the  sign  of  these  z-scores  were 
reflected  to  properly  indicate  which  group  had  the  advantage  before  descriptive  statistics  were 
calculated.) 


Median 

Mean 

Minimum 

Maximum 

Range 

White  minus 

ASVAB 

.727 

.588 

.023 

1.106 

1.083 

Black 

ECAT 

.656 

.615 

.445 

.729 

.284 

White  minus 

ASVAB 

.367 

.289 

-.396 

.829 

1.225 

Asian 

ECAT 

.139 

.144 

-.100 

.400 

.508 

White  minus 

ASVAB 

.314 

.299 

-.017 

.638 

.655 

Hispanic 

ECAT 

.116 

.137 

.026 

.248 

.222 

Black  minus 

ASVAB 

-.246 

-.299 

-.566 

.020 

.586 

Asian 

ECAT 

-.443 

-.470 

-.756 

-.085 

.671 

Black  minus 

ASVAB 

-.320 

-.288 

-.486 

-.001 

.467 

Hispanic 

ECAT 

-.539 

i 

<1 

oo 

-.615 

-.197 

.418 

Asian  minus 

ASVAB 

-.002 

.011 

-.224 

.379 

.603 

Hispanic 

ECAT 

.087 

-.007 

-.266 

.189 

.455 

Paying  close  attention  to  the  sign  of  the  z-scores  reveals  several  facts  about  the  test  batteries 
and  ethnic  groups.  Whites  outperform  all  ethnic  groups  on  both  batteries,  but  the  median  ethnic 
difference  is  higher  for  ASVAB  than  ECAT  in  all  cases.  All  ethnic  groups  outperformed  Blacks 
on  both  test  batteries.  While  the  ECAT  battery  produces  a  smaller  median  advantage  for  Whites 
over  the  other  ethnic  groups  in  comparison  with  the  ASVAB,  ECAT  does  produce  slightly  larger 
differences  among  Black,  Asian,  and  Hispanic  groups  relative  to  one  another  (and  the  ASVAB). 
That  is,  ECAT  reduces  the  advantage  of  the  majority  group  (White)  but  it  increases  the 
differentiation  among  minority  groups.  However,  concerns  over  adverse  impact  and  group 
differences  invariably  focus  only  on  majority-minority  comparisons. 


In  terms  of  individual  tests  within  the  ASVAB,  clearly  the  largest  consistent  differences 
appear  for  Auto-Shop  Information  where  Whites  have  an  advantage  ranging  from  1.106  to  .638 
standard  deviation  units  over  the  other  groups.  The  ASVAB  math  tests  provided  the  largest 
advantage  for  Asians  over  Black  (Arithmetic  Reasoning)  and  Hispanic  (Math  Knowledge) 
groups.  The  largest  Hispanic  advantage  over  Blacks  was  also  for  the  Auto-Shop  Information 
test.  For  the  individual  ECAT  tests,  no  one  test  consistently  differentiated  any  of  the  groups 
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across  the  six  comparisons.  Indeed,  six  different  tests  produced  the  largest  z-score  difference  in 
the  six  comparisons. 


More  detailed  information  follow  these  pages.  The  next  two  pages  contain  the  variable 
names  and  descriptions  used  in  the  analyses.  The  variables  include  years  of  education, 
educational  level,  and  dominant  language  as  demographics,  then  test  scores  from  the  ASVAB 
and  ECAT  batteries.  Following  that  are  the  six  pairwise  group  comparison  analyses  (White- 
Black,  White-Asian,  White-Hispanic,  Black-Asian,  Black-Hispanic,  Asian-Hispanic).  Each 
group  comparison  is  contained  in  a  two-page  table.  The  ASVAB  variables  included  the  ten 
subtest  standard  scores,  AFQT,  and  VE.  For  the  ECAT  tests,  at  least  the  following  variables 
were  looked  at:  time  spent  in  instructions,  time  spent  on  the  test,  average  item  response  times, 
proportion  of  items  attempted,  and  proportion  correct.  Generally,  the  results  converge:  if  a 
group  does  less  well  on  proportion  correct,  they  spent  more  time  in  the  instructions  and  in  the 
test,  had  longer  item  response  times,  and  when  there  was  variation  in  the  number  of  items 
attempted,  they  attempted  fewer.  The  nine  ECAT  scores  that  should  be  focused  on  are: 
IDPCOR  (Integrating  Details  Proportion  Correct),  MCPCOR  (Mental  Counters  Proportion 
Correct),  ORPCOR  (Spatial  Orientation  Proportion  Correct),  SMPDCOR  (Sequential  Memory 
Proportion  of  Digits  Correct),  SRPCOR  (Spatial/Figural  Reasoning  Proportion  Correct), 
AOPCOR  (Assembling  Objects  Proportion  Correct),  TIDDT  (Target  Identification  Decision 
Time),  T1MN  (One-Hand  Tracking  Mean  1000*Log(l+RMS);  a  distance  off-target  measure, 
i.e.,  an  error  score),  and  T2MN  (Two-Hand  Tracking  error  score). 


The  remaining  pages  of  this  letter  report  consist  of: 

A.  List  of  the  variables  and  their  abbreviations  used  in  the  analyses  (2  pages,  F4-F5) 

B.  White  vs.  Black  comparisons  (2  pages,  F6-F7) 

C.  White  vs.  Asian  comparisons  (2  pages,  F8-F9) 

D.  White  vs.  Hispanic  comparisons  (2  pages,  F10-F1 1) 

E.  Black  vs.  Asian  comparisons  (2  pages,  F12-F13) 

F.  Black  vs.  Hispanic  comparisons  (2  pages,  F14-F15) 

G.  Asian  vs.  Hispanic  comparisons  (2  pages,  F16-F17) 
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Variable 

Variable  Description 

SMIT 

SMTT 

SMRT 

SMPIAT 

SMPICOR 

SMPDCOR 

SM  INSTRUCTION  TIME  (IN  SECONDS) 

SM  TESTING  TIME  (IN  SECONDS) 

SM  AVERAGE  ITEM  RESPONSE  LATENCY 

SM  PROPORTION  OF  DIGITS  ATTEMPTED 

SM  PROPORTION  WITH  ALL  5  DIGITS  CORRECT 

SM  PROPORTION  DIGITS  ENTERED  CORRECTLY 

SRIT 

SRTT 

SRRT 

SRPAT 

SRPCOR 

SR  INSTRUCTION  TIME  (IN  SECONDS) 

SR  TESTING  TIME  (IN  SECONDS) 

SR  ARITHMETIC  MEAN  ITEM  RESPONSE  LATENCY 

SR  PROPORTION  OF  TEMS  ATTEMPTED 

SR  PROPORTION  OF  TEST  ITEMS  CORRECT 

AOIT 

AOTT 

AORT 

AOPAT 

AOPCOR 

AO  INSTRUCTION  TIME  (IN  SECONDS) 

AO  TESTING  TIME  (IN  SECONDS) 

AO  ARITHMETIC  MEAN  ITEM  RESPONSE  LATENCY 

AO  PROPORTION  OF  ITEMS  ATTEMPTED 

AO  PROPORTION  OF  ALL  ITEMS  CORRECT 

TIDIT 

TIDTT 

TIDAT 

TIDCOR 

TIDDT 

TIDMT 

TID  INSTRUCTION  TIME  (IN  SECONDS) 

TID  TESTING  TIME  (IN  SECONDS) 

TARGET  ID  NUMBER  OF  VALID  ATTEMPTS 

TARGET  ID  NUMBER  OF  VALID  ITEMS  CORRECT 
TARGET  ID  MEAN  CLIPPED  DECISION  RTS 

TARGET  ID  MEDIAN  VALID  MOVEMENT  TIME 

T1IT 

T1TT 

T1NAT 

T1MN 

T1RMS 

T1SD 

TR1  INSTRUCTION  TIME  (IN  SECONDS) 

TR1  TESTING  TIME  (IN  SECONDS) 

TRACK  1  NUMBER  VALID  ITEM  ATTEMPTS 

TRACK1  MEAN  1000*LOG(1+RMS(ATIEMPTED)) 
TRACK  1  AVERAGE  RMS  DISTANCE  OFF  TARGET 
TRACK1  SD  OF  1 000*LOG  ( 1  +RMS( ATTEMPTED)) 

T2IT 

T2TT 

T2NAT 

T2MN 

T2RMS 

T2SD 

TR2  INSTRUCTION  TIME  (IN  SECONDS) 

TR2  TESTING  TIME  (IN  SECONDS) 

TRACK2  NUMBER  VALID/SCORED  ITEM  ATTEMPTS 
TRACK2  MEAN  1 000*LOG(  1 +RMS(ATTEMPTED)) 
TRACK2  AVERAGE  RMS  DISTANCE  OFF  TARGET 
TRACK2  SD  OF  1 000*LOG  ( 1  +RMS( ATTEMPTED)) 
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Appendix  G 

Criterion  Data  Editing  and  Outlier  Detection 
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Criterion  Data  Editing  and  Outlier  Detection 

Criteria  were  of  two  types:  (1)  Final  School  Grades  (FSG)  and  other  test  scores  (such  as  the 
FAA  exam  or  the  AFPT70  typing  test)  supplied  by  the  training  school,  and  (2)  Composites  of 
more  elementary  measures  of  classroom  or  shop/laboratory  performance.  For  the  latter,  a 
contractor  (RGI  Inc.)  collected  vast  amounts  of  detailed  records  of  homework,  quizzes,  tests 
scores,  simulator  performance  measures,  etc.  Based  on  factor  analyses,  they  formed  composites  of 
variables  loading  on  the  corresponding  factors  (Kieckhaefer,  et  al.,  1992).  In  addition,  some 
composites  were  constructed  on  rational  grounds,  e.g.  FSG  scores  in  the  Army  13F,  Air  Force 
APS,  and  Navy  AC  schools,  were  computed  rather  than  taken  from  school  records. 

One  problem  with  composites  of  internal  measures  is  that  the  curricula  change  frequently,  so 
that  some  students  do  not  take  tests  that  other  students  take.  Another  problem  is  that  most 
students  are  likely  to  miss  a  few  quizzes  or  laboratory  exercises  for  a  variety  of  personal  reasons. 
The  approach  that  was  adopted  for  handling  the  missing  data  was  to  define  a  composite  for  a 
given  student  as  equal  to  the  mean  of  the  component  measures  that  were  present  for  that  student. 
Thus  a  composite  criterion  that  was  supposed  to  consist  of  14  test  scores  would  be  computed  for 
a  student  that  missed  five  exams  and  took,  only  9  of  them,  for  example.  In  most  cases,  the 
component  measures  ranged  from  0  to  100,  but  their  standard  deviations  sometimes  differed  by  a 
factor  of  two.  There  was  no  attempt  to  scale  the  tests  to  have  equal  means  and  standard 
deviations,  nor  to  use  regression  estimates  for  the  missing  values. 

The  criteria  were  subjected  to  a  very  careful  review  and  multi-stage  editing  process.  The  first 
step  was  to  run  a  regression  analysis  for  each  criterion  against  all  10  ASVAB  tests  plus  9  ECAT 
tests  (19  predictors).  For  each  sample  point,  a  DFFITS  measure  of  influence  was  computed,  as 
described  by  Belsley,  Kuh,  and  Welsch  (1980).  The  authors  estimate  the  standard  deviation  of 
DFFITS  to  be  JpTn  and  recommend  a  cutoff  of  2  standard  deviations  for  selecting  influential 

observations  for  further  examination.  In  the  ECAT  sample,  this  rule  would  have  produced  far  too 
many  cases  for  study;  instead  we  selected  cases  whose  DFFITS  values  were  ±4  standard 
deviations  for  further  study. 

On  the  first  pass,  we  simply  tried  omitting  all  such  outliers  from  the  analysis  and  compared 
the  results  with  the  first  regression  analyses.  The  effects  of  deleting  the  high  influence  cases  were 
sometimes  quite  large,  e.g.  the  multiple  correlation  for  the  Army’s  13F  Final  School  Grade  went 
from  .415  to  .597  and  the  standard  error  of  prediction  on  a  Iackknife  cross-validation  went  from 
6.432  to  3.452  when  only  5  of  831  cases  were  deleted.  Similarly  large  changes  were  found  for 
13F  Firing,  11H(A)  EVT2TO,  ATC  BLK5A,  AE  SUM2,  and  RM  FSG.  Results  with  the  outliers 
removed  were  presented  by  Wolfe(1993,  November). 
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Next,  we  began  an  investigation  into  why  the  outliers  occurred.  The  approach  was  to  try  to 
find  general  principles  or  rules  for  excluding  troublesome  cases,  and  to  retain  outliers  in  the  data 
base  unless  a  clear-cut  rule  could  be  found  for  deleting  them.  The  first  explanation  that  turned  up 
was  scoring  error.  Several  types  of  programming  errors  were  detected  and  corrected: 

1.  Missing  test  or  performance  scores  were  treated  as  if  they  were  zeros  when  summing  up 
criterion  composites.  This  occurred  in  all  criteria  for  1 1H,  13F,  and  possibly  some  others. 

2.  Test  scores  that  were  supposed  to  be  in  the  composite  were  omitted  (AE  and  ET). 

3.  Variables  that  were  not  supposed  to  be  in  the  composite  were  included  (AE  and  ET). 

4.  Miscellaneous  programming  errors  with  unknown  effects,  if  any.  These  included  hanging 
DO  loops  without  END  statements  (AV  and  ET),  attempts  to  compare  alphabetic  strings 
with  numeric  values  (AV),  and  defining  an  array  of  variables  with  the  same  variable  twice 
(OS). 

After  rescoring  the  data,  the  regression  analyses  for  outlier  detection  were  run  again.  The 
data  were  examined  again  for  possible  causes  of  strange  behavior.  A  large  percentage  (sometimes 
all)  of  the  outliers  came  from  students  who  dropped  out  of  school  either  because  of  academic 
failure  or  because  of  administrative  reasons,  such  as  bereavement,  illness,  AWOL,  personality 
difficulties,  alcohol,  drugs,  or  disciplinary  reasons.  The  Navy  sample  included  66  academic 
dropouts  and  314  administrative  dropouts.  However,  no  information  on  student  status  was 
available  for  any  Army  or  Air  Force  schools,  or  for  the  Navy  FC  or  OS  schools. 

The  major  reason  that  dropouts  had  atypical  scores  was  that  their  data  were  incomplete.  The 
administrative  drops  sometimes  had  above-average  scores  on  some  criteria,  and  very  deviant 
scores  on  others.  Some  of  them  had  FSGs  in  their  records,  but  based  on  only  the  part  of  the 
curriculum  that  they  completed.  Different  schools  apparently  had  different  policies  for  computing 
these  grades:  sometimes  they  were  missing,  sometimes  they  were  quite  high  (evidently  means  of 
tests  completed),  and  sometimes  they  were  very  low  (possibly  assigning  zeros  to  missing  test 
scores).  Even  where  no  dropout  codes  were  available,  a  frequent  characteristic  of  outliers  was 
very  incomplete  data  regarding  test,  or  performance  scores.  A  composite  criterion  that  was 
supposed  to  be  the  mean  of  10-14  measures  might  contain  only  1  or  2  of  them,  for  example. 

Another,  but  related,  cause  of  outliers  was  non-normal  criterion  distributions  arising  from 
binary  (Pass/Fail)  component  scores.  For  example,  in  the  Air  Force  ATC  school,  several  criteria 
were  defined  as  the  difference  between  a  binary  performance  standard  score  and  a  standardized 
measure  of  time  on  course  section.  In  other  cases,  distributional  problems  arose  when  the  mean 
of  4  to  12  binary  measures  was  reduced  to  the  mean  of  only  1  or  2,  due  to  incomplete  data. 
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To  improve  the  quality  of  the  data,  the  administrative  school  dropouts  were  all  deleted  from 
the  ECAT  data  base.  Next,  the  number  of  components  going  into  each  composite  was  tabulated 
for  each  student.  (These  counts  were  labeled  N_FSG,  N_ADV,  etc.)  Rules  were  formulated  that 
eliminated  most  of  the  remaining  outliers  as  follows: 

APS  Drop  cases  with  N_FSG  <  3  out  of  6  tests. 

AC  Drop  cases  with  N_PRF  <  2  out  of  4  tests. 

A  V  Drop  cases  if  N_ADV  +  N_PERF  <  2  out  of  8  tests. 

ET  Drop  cases  where  MEAN(N_PRF1  ,  N_PRF2)  <  1  out  of  10  or  14  tests, 

respectively. 

FC  Drop  37  cases  with  N_RADAR  =  0. 

Four  outliers  appeared  to  be  associated  with  previously  undetected  problems  with  their 
ECAT  tests.  In  ECAT  a  "jump"  is  defined  as  a  fast  response  to  a  difficult  test  item.  In  AMS 
school,  one  examinee  had  26  jumps  in  the  Mental  Counters  test  and  got  27%  of  the  items  correct. 
In  OS  school,  one  outlier  had  21  jumps  in  the  Figural  Reasoning  test  with  only  16.7%  correct.  In 
ATC  and  APS  schools,  two  outliers  had  high  jumps  and  low  scores  on  the  Mental  Counters  test. 
These  examinees  were  obviously  not  trying,  but  were  pressing  keys  at  random.  Further 
examination  of  the  the  entire  ECAT  sample  showed  that  of  19  cases  with  more  than  6  jumps  on 
Figural  Reasoning,  none  scored  higher  than  50%  on  the  test.  On  Mental  Counters,  none  of  the  73 
cases  that  had  more  than  15  jumps  scored  higher  than  50%  correct.  Because  some  high  jump 
cases  scored  above  chance,  it  was  difficult  to  formulate  a  general  principle  for  screening  outliers 
that  did  not  also  exclude  too  many  legitimate  cases.  Thus  the  four  outliers  previously  identified 
were  retained  in  the  ECAT  sample. 

In  a  few  remaining  outliers,  the  reasons  for  their  atypical  behavior  could  not  be  determined. 
For  example,  in  11H(B),  the  EVT1TO,  EVT2TO,  and  EVT3TO  scores  were  each  the  sum  of 
scores  on  ten  "shots".  The  individual  shots  were  scored  on  a  scale  from  0  to  100.  One  student 
received  scores  of  0  on  all  ten  of  his  EVT1TO  shots  and  on  eight  of  his  EVT2TO  shots.  He  was 
retained  in  the  ECAT  sample,  because  there  was  no  way  of  knowing  why  he  behaved  as  he  did. 
Perhaps  he  had  some  basic  misconception  about  the  task,  perhaps  he  forgot  his  glasses  that  day, 
perhaps  the  equipment  malfunctioned,  etc. 

On  the  final  analyses,  the  policy  was  to  exclude  only  those  cases  (and  all  such  cases),  that 
violated  some  clear-cut  principle,  regardless  of  whether  they  were  outliers  or  not.  This  policy 
resulted  in  excluding  many  more  non-outliers  than  outliers  (e.g.  the  314  administrative  dropouts) 
while  retaining  some  outliers  for  which  it  was  difficult  to  formulate  a  rationale  for  deleting  them. 

It  should  be  noted  that  in  the  majority  of  academic  failures,  no  Final  School  Grades  were 
available,  and  we  did  not  choose  to  impute  scores  to  these  failures.  However,  knowledge  and 
performance  test  averages  were  present  in  most  cases. 
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As  a  final  check,  regression  analyses  were  run  with  and  without  the  remaining  outliers  to 
determine  the  magnitudes  of  the  effects  on  multiple  correlations.  Tables  G-l  to  G-3  show,  with 
some  exceptions,  that  the  remaining  outliers  have  little  effect  on  the  multiple  correlations,  and 
even  less  on  the  differences  between  ASVAB  and  ASVAB  +  ECAT  multiple  correlations,  which 
are  the  incremental  validities. 


Table  G-l 


Effect  of  Dropping  High  Influence  Cases  on  Multiple  Correlations 

All  Army  Criteria 


School 

Criterion 

Full 

N 

Outliers 

Dropped 

ASVAB 

R  R-Change 

ASVAB+ECAT 

R  R-Change 

Incremental 

Validity  Change 

11H(A)1 

TO  ALL 

554 

6 

.242 

.002 

.296 

-.023 

-.026 

11H(A)2 

EVT1TO 

556 

1 

.316 

.012 

.365 

.014 

.001 

11H(A)3 

EVT2TO 

555 

0 

.242 

.000 

.294 

.000 

.000 

11H(A)4 

EVT3TO 

550 

1 

.294 

-.007 

.347 

-.006 

.001 

11H(A)5 

EVTSUM 

546 

0 

.321 

.000 

.382 

.000 

.000 

11H(A)6 

TO_l 

542 

4 

.210 

-.004 

.274 

-.003 

.001 

11H(B)1 

TOALL 

320 

4 

.313 

-.061 

.364 

-.052 

.009 

1 1H(B)2 

EVT1TO 

320 

0 

.291 

.000 

.416 

.000 

.000 

11H(B)3 

EVT2TO 

319 

1 

.312 

.006 

.442 

-.005 

-.011 

11H(B)4 

EVT3TO 

319 

1 

.234 

-.007 

.301 

-.009 

-.003 

11H(B)5 

EVTSUM 

316 

1 

.330 

.016 

.456 

.001 

-.015 

1 1H(B)6 

TO_l 

319 

0 

.144 

.000 

.327 

.000 

.000 

11H(B)7 

TO_2 

320 

0 

.172 

.000 

.311 

.000 

.000 

11H(B)8 

TO_3 

319 

2 

.176 

-.013 

.340 

.009 

.022 

11H(B)9 

ITVTOW 

318 

0 

.154 

.000 

.354 

.000 

.000 

13F1 

FSG 

821 

0 

.544 

.000 

.598 

.000 

.000 

13F2 

MPRAD 

821 

2 

.513 

.006 

.591 

-.002 

-.009 

13F3 

FIRING 

821 

0 

.444 

.000 

.472 

.000 

.000 

19K1 

COMM 

1158 

19 

.080 

.005 

.142 

-.009 

-.015 

19K2 

WEAPON 

1325 

9 

.187 

-.005 

.211 

.000 

.005 

19K3 

LANDNAV 

1192 

15 

.175 

-.012 

.198 

-.011 

.001 

19K4 

LOADER 

1313 

2 

.066 

.007 

.092 

.005 

-.003 

19K5 

MAINT 

1329 

6 

.128 

.017 

.163 

.004 

-.013 

19K6 

NBC 

1313 

11 

.119 

-.023 

.142 

-.013 

.010 

19K7 

AVERAGE 

1106 

7 

.208 

-.003 

.227 

.008 

.010 
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Table  G-2 


Effect  of  Dropping  High  Influence  Cases  on  Multiple  Correlations 

All  Air  Force  Criteria 


School 

Criterion 

Full 

N 

Outliers 

Dropped 

ASVAB 

R  R-Change 

ASVAB+ECAT 

R  R-Change 

Incremental 

Validity  Change 

APSl 

FSG 

446 

1 

.545 

.003 

.585 

.005 

.003 

APS2 

ZHRS 

446 

2 

.424 

.011 

.487 

.004 

-.008 

APS3 

AFPT70 

432 

3 

.294 

-.000 

.422 

.000 

.001 

APS5 

BYPAS1 

369 

2 

.296 

.024 

.394 

.002 

-.022 

APS6 

FINAL 

369 

2 

.296 

.024 

.394 

.002 

-.022 

APS7 

DWPM 

357 

3 

.213 

.037 

.244 

.036 

-.001 

ATC1 

FSG 

484 

2 

.403 

.008 

.451 

.009 

.000 

ATC2 

BLK2 

349 

0 

.374 

.000 

.421 

.000 

.000 

ATC3 

BLK3A 

529 

9 

.153 

.013 

.229 

.004 

-.009 

ATC4 

BLK3B 

217 

3 

.165 

.004 

.367 

-.035 

-.039 

ATC5 

BLK5A 

500 

6 

.267 

.002 

.374 

-.009 

-.011 

ATC6 

BLK5B 

495 

3 

.216 

-.021 

.274 

-.023 

-.002 

ATC7 

FAA 

536 

2 

.490 

-.009 

.540 

-.005 

.003 

ATC(A)1 

FSG 

200 

0 

.389 

.000 

.471 

.000 

.000 

ATC(A)2 

BLK3A 

221 

3 

.279 

.029 

.348 

.013 

-.016 

ATC(A)3 

BLK3B 

217 

3 

.165 

.004 

.367 

-.035 

-.039 

ATC(A)4 

BLK5A 

205 

0 

.322 

.000 

.438 

.000 

.000 

ATC(A)5 

BLK5B 

204 

1 

.214 

-.007 

.276 

-.024 

-.017 

ATC(A)6 

FAA 

251 

2 

.508 

-.007 

.547 

-.008 

-.001 

ATC(B)1 

FSG 

284 

1 

.449 

.011 

.485 

.015 

.004 

ATC(B)2 

BLK2 

349 

0 

.374 

.000 

.421 

.000 

.000 

ATC(B)3 

BLK3A 

308 

5 

.195 

-.037 

.296 

-.022 

.015 

ATC(B)4 

BLK5A 

295 

5 

.312 

.017 

.414 

.037 

.020 

ATC(B)5 

BLK5B 

291 

4 

.264 

-.016 

.336 

-.006 

.010 

ATC(B)6 

FAA 

285 

0 

.485 

.000 

.529 

.000 

.000 

Table  G-3 


Effect  of  Dropping  High  Influence  Cases  on  Multiple  Correlations 

All  Navy  Criteria 


School 

Criterion 

Full 

N 

Outliers 

Dropped 

ASVAB 

R  R-Change 

ASVAB+ECAT 

R  R-Change 

Incremental 

Validity  Change 

AC1 

FSG 

72 

0 

.627 

.000 

.659 

.000 

.000 

AC2 

PERF 

76 

1 

.330 

.028 

.498 

.095 

.067 

AC3 

FAA 

76 

0 

.454 

.000 

.562 

.000 

.000 

AE1 

FSG 

278 

2 

.489 

.008 

.550 

.001 

-.007 

AE2 

SUM2 

273 

3 

.440 

-.008 

.498 

-.015 

-.007 

AMS1 

FSG 

244 

1 

.599 

-.012 

.604 

-.009 

.003 

AMS2 

PERF 

244 

2 

.393 

-.005 

.437 

-.009 

-.004 

AOl 

FSG 

234 

0 

.504 

.000 

.537 

.000 

.000 

A02 

PRACTL 

229 

3 

.343 

.018 

.393 

.010 

-.008 

AVI 

FSG 

544 

0 

.517 

.000 

.539 

.000 

.000 

AV2 

BSCAV 

192 

0 

.531 

.000 

.571 

.000 

.000 

AV3 

ADVAV 

192 

0 

.358 

.000 

.404 

.000 

.000 

AV4 

PERFORM 

352 

4 

.379 

-.028 

.417 

-.030 

-.002 

EMI 

FSG 

797 

0 

.451 

.000 

.471 

.000 

.000 

EM2 

PHASE  1 

797 

0 

.474 

.000 

.485 

.000 

.000 

EN1 

FSG 

750 

1 

.584 

.013 

.593 

.014 

.001 

ET1 

FSG 

86 

0 

.509 

.000 

.629 

.000 

.000 

ET2 

FSG2 

86 

1 

.504 

.035 

.574 

.015 

-.020 

ET3 

PERF 

86 

2 

.482 

.044 

.585 

.077 

.033 

FC1 

FSG 

778 

0 

.499 

.000 

.536 

.000 

.000 

FC2 

RADAR 

780 

2 

.345 

-.005 

.388 

-.009 

-.003 

GM1 

FSG 

420 

1 

.428 

.005 

.465 

.010 

.005 

GM2 

HALF1 

420 

1 

.442 

.006 

.496 

.011 

.006 

GM3 

HALF2 

397 

2 

.458 

.003 

.479 

.007 

.004 

MM1 

FSG 

801 

0 

.402 

.000 

.438 

.000 

.000 

OS1 

FSG 

713 

2 

.565 

.008 

.588 

.009 

.001 

OS2 

WRIT 

815 

2 

.478 

.001 

.496 

.000 

-.001 

OS3 

PERF 

815 

1 

.523 

-.001 

.566 

-.001 

.000 

RM1 

FSG 

277 

0 

.536 

.000 

.592 

.000 

.000 

RM2 

PHASE3 

277 

0 

.420 

.000 

.467 

.000 

.000 
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Appendix  I 


Test  Validities  and  Incremental  Validities  for  All  Criteria 


i-o 


Table  1-1 


ECAT  Incremental  Validities  for  all  Army  Criteria 


Uncorrected  Multiple  R _  Corrected  Multiple  R 


School 

Criterion 

Sample 

Size 

ASVAB 

ASVAB 

+ECAT 

Percent 

Variance 

Probability 
°f  ^6,N-n 

ASVAB 

Increase 

Percent 

11H(A)1 

TOALL 

554 

.242 

.275 

1.807 

1.40X10’1 

.272 

.013 

4.7 

1 1H(A)2 

EVT1TO 

556 

.316 

.362 

3.644 

3.59x10’’ 

.404 

.030 

7.4  ** 

11H(A)3 

EVT2TO 

555 

.242 

.286 

2.549 

3.46x1  O’2 

.273 

.034 

12.6* 

1 1H(A)4 

EVT3TO 

550 

.294 

.329 

2.438 

4.48X10’2 

.365 

.019 

5.3* 

11H(A)5 

EVTSUM 

546 

.321 

.373 

4.119 

1.53x10’’ 

.392 

.036 

9.2  ** 

11H(A)6 

TO_l 

542 

.210 

.269 

3.031 

1.52X10’2 

.240 

.046 

19.1  * 

11H(B)1 

TOALL 

320 

.313 

.335 

1.629 

5.53x10’’ 

.372 

.000 

0.1 

1 1H(B)2 

EVT1TO 

320 

.291 

.398 

8.765 

2.62x1  O'4 

.305 

.089 

29.1  ** 

1 1H(B)3 

EVT2TO 

319 

.312 

.434 

11.171 

1.50xl0’s 

.389 

.095 

24.5  ** 

11H(B)4 

EVT3TO 

319 

.234 

.292 

3.308 

1.29x10’’ 

.243 

.029 

11.9 

11H(B)5 

EVTSUM 

316 

.330 

.446 

11.216 

1.64xl0’5 

.382 

.091 

23.7  ** 

11H(B)6 

TO_l 

319 

.144 

.317 

8.843 

2.47x10^ 

.014 

.269 

1979  ** 

11H(B)7 

TO_2 

320 

.172 

.309 

7.313 

1.48x10’’ 

.093 

.162 

173.6  ** 

1 1H(B)8 

TO_3 

319 

.176 

.329 

8.710 

2.90x10"* 

.054 

.225 

418.8  ** 

11H(B)9 

ITVTOW 

318 

.154 

.350 

11.203 

1.51x10’’ 

.075 

.237 

316.3  ** 

13F1 

FSG 

821 

.544. 

.597 

9.483 

9.81X10’14 

.790 

.024 

3.0** 

13F2 

MPRAD 

821 

.513 

.590 

12.950 

<1.0x10’” 

.809 

.040 

13F3 

FIRING 

821 

.444 

.466 

2.507 

2.82x10’’ 

.730 

.007 

1.0** 

19K1 

COMM 

1158 

.080 

.135 

1.208 

3.28X10’2 

.000 

.071 

* 

19K2 

WEAPON 

1325 

.187 

.205 

.738 

1.41x10’’ 

.198 

.006 

3.2 

19K3 

LANDNA 

1192 

.175 

.192 

.617 

2.99x10’’ 

.190 

.005 

2.5 

19K4 

LOADER 

1313 

.066 

.087 

.330 

6.40X10’’ 

.000 

.000 

. 

19K5 

MAINT 

1329 

.128 

.154 

.767 

1.23x10’’ 

.109 

.011 

10.4 

19K6 

NBC 

1313 

.119 

.136 

.429 

4.75x10’’ 

.128 

.001 

0.5 

19K7 

AVERAG 

1106 

.208 

.226 

.834 

1.70x10’’ 

.392 

.006 

1.7 

Percent  Variance  = 


100 


AR2 


1  -R 


2 

ASVAB+ECAT 


*  p  <  .05.  **  p  <  .01. 
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Table  1-2 


ECAT  Incremental  Validities  for  all  Air  Force  Criteria 


Uncorrected  Multiple  R _  Corrected  Multiple  R 


School 

Criterion 

Sample 

Size 

ASVAB 

ASVAB 

+ECAT 

Percent 

Variance 

Probability 
°f  ^6,N-n 

ASVAB 

Increase 

Percent 

APSl 

FSG 

446 

.545 

.581 

6.233 

2. 17x1  O'4 

.828 

.012 

1.5** 

APS2 

ZHRS 

446 

.424 

.476 

6.059 

2.94x10"* 

.680 

.023 

3.4  ** 

APS3 

AFPT70 

432 

.294 

.404 

9.129 

2.28X10-* 

.388 

.079 

20.4  ** 

ATC1 

FSG 

484 

.403 

.445 

4.540 

1.98x10-’ 

.727 

.020 

2.7  ** 

ATC2 

BLK2 

349 

.374 

.410 

3.373 

8.60x1 0"2 

.660 

.009 

1.4 

ATC3 

BLK3A 

529 

.153 

.217 

2.481 

4.99X10"2 

.267 

.062 

23.1  * 

ATC4 

BLK3B 

217 

.165 

.341 

10.057 

3.61X103 

.000 

.368 

** 

ATC5 

BLK5A 

500 

.267 

.359 

6.618 

2.45x1 0"5 

.494 

.090 

18.2  ** 

ATC6 

BLK5B 

495 

.216 

.263 

2.422 

7.45x1  O'2 

.444 

.031 

7.0 

ATC7 

FAA 

536 

.490 

.523 

4.600 

6.64x10^ 

.757 

.013 

1.7  ** 

ATC(A)1 

FSG 

200 

.389 

.464 

8.205 

2.37x1 0"2 

.680 

.043 

6.3* 

ATC(A)2 

BLK3A 

221 

.279 

.314 

2.339 

5.75X10'1 

.302 

.000 

0.1 

ATC(A)3 

BLK3B 

217 

.165 

.341 

10.057 

3.61x10-’ 

.000 

.368 

** 

ATC(A)4 

BLK5A 

205 

.322 

.404 

7.127 

4.18X10"2 

.614 

.079 

12.9* 

ATC(A)5 

BLK5B 

204 

.214 

.254 

2.008 

7.10X10"' 

.276 

.013 

4.6 

ATC(A)6 

FAA 

251 

.508 

.543 

5.179 

6.39x1 0"2 

.744 

.011 

1.5 

ATC(B)1 

FSG 

284 

.449 

.480 

3.788 

1.25x10-' 

.758 

.009 

1.2 

ATC(B)2 

BLK2 

349 

.374 

.410 

3.373 

8.60X10’2 

.660 

.009 

1.4 

ATC(B)3 

BLK3A 

308 

.195 

.284 

4.627 

3. 92x1  O'2 

.208 

.142 

68.4* 

ATC(B)4 

BLK5A 

295 

.312 

.408 

8.316 

1.04x10"’ 

.450 

.100 

22.2  ** 

ATC(B)5 

BLK5B 

291 

.264 

.330 

4.376 

6.61X10’2 

.541 

.041 

7.6 

ATC(B)6 

FAA 

285 

.485 

.516 

4.160 

8.83X10'2 

.728 

.015 

2.1 

Percent  Variance  = 


100 


A R2 


1  -R 


2 

ASVAB+ECAT 


*  p  <  .05. 


**  p  <  .01. 


Table  1-3 


ECAT  Incremental  Validities  for  all  Navy  Criteria 


Uncorrected  Multiple  R  _  Corrected  Multiple  R 


School 

Criterion 

Sample 

Size 

ASVAB 

ASVAB 

+ECAT 

Percent 

Variance 

Probability 

°f  ^6,N- 17 

ASVAB 

Increase 

Percent 

AC1 

FSG 

72 

.627 

.649 

4.978 

8.37x10*' 

.839 

.000 

0 

AC2 

PERF 

76 

.330 

.460 

13.033 

2.80x10"' 

.381 

.149 

39.2 

AC3 

FAA 

76 

.454 

.540 

11.968 

3.31x10*' 

.551 

.043 

7.8 

AE1 

FSG 

278 

.489 

.542 

7.810 

3.04x1 0*3 

.659 

.023 

3.5  ** 

AE2 

SUM2 

273 

.440 

.487 

5.808 

2.39x1 0"2 

.608 

.022 

3.7* 

AMS1 

FSG 

244 

.599 

.602 

.555 

9.73x10*’ 

.848 

.000 

0 

AMS2 

PERF 

244 

.393 

.431 

3.892 

1.89x10*' 

.650 

.016 

2.4 

AOl 

FSG 

234 

.504 

.522 

2.434 

5.10x10*' 

.717 

.005 

0.7 

A02 

PRACTL 

229 

.343 

.374 

2.652 

4.69x10*' 

.490 

.010 

2.1 

AVI 

FSG 

544 

.517 

.536 

2.772 

2.49X10*2 

.810 

.005 

0.7* 

AV2 

BSCAV 

192 

.531 

.565 

5.494 

1.49x10*’ 

.844 

.008 

0.9 

AV3 

ADVAV 

192 

.358 

.402 

4.003 

3.26x10*' 

.694 

.009 

1.3 

AV4 

PERFOR 

352 

.379 

.409 

2.853 

1.48x10*' 

.673 

.016 

2.4 

EMI 

FSG 

797 

.451 

.459 

.864 

3.47x10*' 

.687 

.000 

0 

EM2 

PHASE  1 

797 

.474 

.482 

.950 

2.86x10*' 

.729 

.001 

0.1 

EN1 

FSG 

750 

.584 

.588 

.721 

5.09x10*’ 

.763 

.000 

0 

ET1 

FSG 

86 

.509 

.603 

16.470 

9.42X10'2 

.805 

.043 

5.3 

ET2 

FSG2 

86 

.504 

.566 

9.738 

3.60x10*' 

.813 

.027 

3.3 

ET3 

PERF 

86 

.482 

.574 

14.533 

1.41X10*' 

.735 

.075 

10.2 

FC1 

FSG 

778 

.499 

.528 

4.180 

2.28X10*5 

.828 

.010 

1.2  ** 

FC2 

RADAR 

780 

.345 

.381 

3.053 

7.93x10-* 

.733 

.016 

2.1  ** 

GM1 

FSG 

420 

.428 

.454 

2.911 

7.10X10*2 

.731 

.004 

0.6 

GM2 

HALF1 

420 

.442 

.478 

4.273 

9.48x10"’ 

.762 

.008 

1.0  ** 

GM3 

HALF2 

397 

.458 

.467 

1.033 

6.87x10*' 

.734 

.000 

0 

MM1 

FSG 

801 

.402 

.425 

2.362 

5.41X10'3 

.557 

.012 

2.2  ** 

OS1 

FSG 

713 

.565 

.582 

2.969 

2.33X10*5 

.804 

.007 

09  ** 

OS2 

WRIT 

815 

.478' 

.489 

1.405 

8.34X10*2 

.756 

.003 

0.4 

OS3 

PERF 

815 

.523 

.564 

6.510 

3.81x10*’ 

.791 

.025 

3.1  ** 

RM1 

FSG 

277 

.536 

.587 

8.796 

1.17X10*3 

.775 

.022 

2.8  ** 

RM2 

PHASE3 

277 

.420 

.464 

4.907 

5.08X10*2 

.702 

.017 

2.4 

Percent  Variance  =  100 - ; - 

1  —  P2 

1  ^  ASVAB + EC  AT 


*  p  <  .05.  **  p  <  .01. 
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Table  1-4 


Incremental  Validities  from  Adding  one  ECAT  test  to  the  ASVAB 
AH  Significant  Criteria  from  Full  Model 


School 

Criterion 

Mental 

Counters 

Sequential 

Memory 

Integrating 

Details 

Assembling 

Objects 

11H(A)2 

EVT1TO 

.012* 

.004 

.008* 

.015** 

1 1H(A)3 

EVT2TO 

.009 

.004 

.001 

.026** 

1 1H(A)4 

EVT3TO 

.006 

.009 

.000 

.025** 

1 1H(A)5 

EVTSUM 

.013* 

.008 

.004 

027** 

1 1H(A)6 

TO_l 

.000 

.000 

.000 

.000 

1 1H(B)2 

EVT1TO 

.000 

037** 

o 

00 

* 

.007 

11H(B)3 

EVT2TO 

.020* 

.036** 

.026** 

.008 

1 1H(B)5 

EVTSUM 

.008 

.024* 

.016* 

.003 

1 1H(B)6 

TO_l 

.084 

.116 

.000 

.078* 

1 1H(B)7 

TO_2 

.000 

.000 

.034* 

.021 

11H(B)8 

TO_3 

.000 

.000 

.009 

.053* 

11H(B)9 

ITVTOW 

.000 

.000 

.006 

.056* 

13F1 

FSG 

.010** 

.009** 

.012** 

.012** 

13F2 

MPRAD 

.019** 

on** 

.023** 

.021** 

13F3 

FIRING 

.002* 

.007** 

.002* 

.002* 

19K1 

COMM 

.000 

.000 

.000 

.000 

APS1 

FSG 

.002 

.006** 

.003* 

.000 

APS2 

ZHRS 

.003* 

.023** 

.000 

.000 

APS3 

AFPT70 

.018** 

.034** 

.025** 

.010* 

ATC1 

FSG 

.015** 

.004 

.001 

.005* 

ATC3 

BLK3A 

.022 

.038* 

.005 

.000 

ATC4 

BLK3B 

.165 

.210** 

.000 

.000 

ATC5 

BLK5A 

.078** 

.019 

.018** 

.027** 

ATC7 

FAA 

.018** 

.001 

.001* 

.004* 

ATC(A)1 

FSG 

.037** 

.015* 

.002 

.002 

ATC(A)3 

BLK3B 

.165 

.210** 

.000 

.000 

ATC(A)4 

BLK5A 

.111** 

.006 

.026* 

.015 

ATC(B)3 

BLK3A 

.058 

.127* 

.006 

.000 

ATC(B)4 

BLK5A 

.060* 

.032 

.014 

.040* 

AC2 

PERF 

.048 

.135* 

.045 

.126* 

AE1 

FSG 

.010** 

.020** 

.019** 

.009* 

AE2 

SUM2 

.008* 

.018** 

.005 

.004 

AVI 

FSG 

.007** 

.002* 

.002* 

.002* 

FC1 

FSG 

.000 

.000 

.001 

.003** 

FC2 

RADAR 

.000 

.005** 

.000 

.001 

GM2 

HALF1 

.007** 

.000 

.001 

.001 

GM3 

HALF2 

.002 

.000 

.000 

.002 

OS1 

FSG 

.007** 

.003* 

.002* 

.002* 

OS3 

PERF 

.017** 

.011** 

.006** 

.010** 

RM1 

FSG 

.004 

.002 

.004 

.000 

1-4 


*  p  <  .05. 


**p<  .01. 


Table  1-4  (continued) 


School 

Criterion 

One-hand 

Tracking 

Two-hand 

Tracking 

Target 

Identification 

Spatial 

Orientaion 

11H(A)2 

EVT1TO 

.013* 

.021** 

.006 

.017** 

11H(A)3 

EVT2TO 

.020* 

.028** 

.024* 

.004 

1 1H(A)4 

EVT3TO 

.014* 

021** 

.003 

.003 

11H(A)5 

EVTSUM 

.019** 

.029** 

.011* 

.010* 

1 1H(A)6 

TO_l 

.036** 

.044** 

.000 

.008 

11H(B)2 

EVT1TO 

.057** 

o 

oo 

* 

* 

.026* 

.038** 

11H(B)3 

EVT2TO 

059** 

.071** 

.021* 

.018* 

11H(B)5 

EVTSUM 

.059** 

.078** 

.021* 

.022** 

1 1H(B)6 

TO_l 

.132* 

.152** 

.000 

.027 

11H(B)7 

TO_2 

.138** 

.160** 

.000 

.014 

11H(B)8 

TO_3 

.201** 

.182** 

.000 

.108** 

11H(B)9 

ITVTOW 

.159** 

.172** 

.000 

.047* 

13F1 

FSG 

.005** 

.003** 

.003* 

.010** 

13F2 

MPRAD 

.003** 

.003** 

.004** 

014** 

13F3 

FIRING 

.006** 

.002* 

.002 

.002* 

19K1 

COMM 

.000 

.000 

.030* 

.000 

APS1 

FSG 

.000 

•  .000 

.000 

.002* 

APS2 

ZHRS 

.000 

.000 

.004* 

.000 

APS3 

AFPT70 

.006 

.028** 

.000 

.004 

ATC1 

FSG 

.006* 

.003 

.008** 

.010** 

ATC3 

BLK3A 

.030* 

.059** 

.000 

.000 

ATC4 

BLK3B 

.000 

.074 

.000 

.091 

ATC5 

BLK5A 

.047** 

.031** 

.012 

.023** 

ATC7 

FAA 

.001 

.003* 

.000 

.007** 

ATC(A)1 

FSG 

.021* 

.014* 

.016* 

.001 

ATC(A)3 

BLK3B 

.000 

.074 

.000 

.091 

ATC(A)4 

BLK5A 

.030 

.015 

.005 

.000 

ATC(B)3 

BLK3A 

.089* 

.076* 

.008 

.029 

ATC(B)4 

BLK5A 

.049** 

.034** 

.023* 

.044** 

AC2 

PERF 

.063 

.000 

.000 

.033 

AE1 

FSG 

.004* 

.000 

.004 

.004* 

AE2 

SUM2 

.000 

.000 

.009* 

.000 

AVI 

FSG 

.000 

.002 

.000 

.001 

FC1 

FSG 

.000 

.001* 

.001* 

.000 

FC2 

RADAR 

.002 

.004* 

.000 

.000 

GM2 

HALF1 

.001 

.001 

.001 

.005** 

GM3 

HALF2 

.000 

.000 

.000 

.001 

OS1 

FSG 

.000 

.001 

.000 

.003* 

OS3 

PERF 

.003* 

.006** 

.000 

.011** 

RM1 

FSG 

.002 

.000 

.011** 

.002 

*  p  <  .05.  **  p  <  .01. 
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Table  1-4  (continued) 


School 

Criterion 

Memory 

Composite 

Spatial 

Composite 

Tracking 

Composite 

Figural 

Reasoning 

11H(A)2 

EVT1TO 

.011* 

.016** 

.020** 

.000 

11H(A)3 

EVT2TO 

.011 

.017* 

.028** 

.002 

11H(A)4 

EVT3TO 

.011* 

.012* 

.021** 

.000 

1 1H(A)5 

EVTSUM 

.015* 

.020** 

.028** 

.001 

11H(A)6 

TO_l 

.000 

.000 

.047** 

.007 

11H(B)2 

EVT1TO 

.022* 

.019* 

.080** 

.000 

11H(B)3 

EVT2TO 

.039** 

.024* 

.076** 

.000 

11H(B)5 

EVTSUM 

.023* 

.013 

.080** 

.000 

11H(B)6 

TO_l 

.126* 

.035* 

.156** 

.000 

11H(B)7 

TO_2 

.000 

.043* 

.168** 

.000 

11H(B)8 

TO_3 

.000 

.050* 

.213** 

.000 

11H(B)9 

ITVTOW 

.004 

.047** 

.185** 

.000 

13F1 

FSG 

.013** 

.017** 

.005** 

.010** 

13F2 

MPRAD 

.021** 

.032** 

.004** 

.016** 

13F3 

FIRING 

.006** 

.003** 

.005** 

.003** 

19K1 

COMM 

.000 

.000 

.000 

.000 

APS1 

FSG 

.006** 

.002* 

.000 

.010** 

APS2 

ZHRS 

.016** 

.000 

.000 

.004* 

APS3 

AFPT70 

.036** 

.024** 

.018** 

.014** 

ATC1 

FSG 

.013** 

.004* 

.005* 

.000 

ATC3 

BLK3A 

.043* 

.000 

.051** 

.015 

ATC4 

BLK3B 

.229** 

.000 

.000 

.294** 

ATC5 

BLK5A 

.062** 

.032** 

.045** 

04i** 

ATC7 

FAA 

on** 

.004** 

.002 

.004* 

ATC(A)1 

FSG 

.036** 

.004 

.021* 

.010 

ATC(A)3 

BLK3B 

.229** 

.000 

.000 

.294** 

ATC(A)4 

BLK5A 

.066** 

.031** 

.027 

.060** 

ATC(B)3 

BLK3A 

.125* 

.000 

.097* 

.019 

ATC(B)4 

BLK5A 

.063* 

.038* 

.049** 

.036 

AC2 

PERF 

.128 

.123 

.025 

.070 

AE1 

FSG 

.021** 

.020** 

.003* 

.009* 

AE2 

SUM2 

019** 

.007 

.000 

.003 

AVI 

FSG 

.007** 

.003* 

.001 

.001 

FC1 

FSG 

.000 

.003** 

.000 

.004** 

FC2 

RADAR 

.003* 

.000 

.004* 

.003 

GM2 

HALF1 

.001 

.001 

.001* 

.004** 

GM3 

HALF2 

.000 

.000 

.000 

.001 

OS1 

FSG 

.007** 

.003** 

.000 

.004** 

OS3 

PERF 

.019** 

.012** 

.005** 

.007** 

RM1 

FSG 

.005 

.002 

.002 

.000 

*p<.05.  **p<.01. 
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Uncorrected  Simple  Validities  for  all  Army  Criteria 
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Mental  Counters  Proportion  Correct  SM  =  Sequential  Memory  Proportion  Correct  ID  =  Integrating  Details  Proportion  Correct 

Assembling  Objects  Proportion  Correct  T1  =  1-Hand  Tracking  Mean  1000*log(l  +  RMS  (Attempted))  T2  =  2-Hand  Tracking  Mean  1000*log(l  +  RMS(Attempted)) 

Figural  Reasoning  Proportion  Correct  SO  =  Spatial  Orientation  Proportion  Correct  TI  =  Target  Identification  Mean  Clipped  Decision  RTs 


Uncorrected  Simple  Validities  for  all  Air  Force  Criteria 
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Assembling  Objects  Proportion  Correct  T1  =  1-Hand  Tracking  Mean  1000*log(l  +  RMS  (Attempted))  T2  =  2-Hand  Tracking  Mean  1000*log(l  +  RMS(Attempted)) 

Figural  Reasoning  Proportion  Correa  SO  =  Spatial  Orientation  Proportion  Correa  TI  =  Target  Identification  Mean  Clipped  Decision  RTs 


Table  1-7 

Uncorrected  Simple  Validities  for  all  Navy  Criteria 


p 

% 

CM 

CO 

CM 

p 

S 

CM 

oo 

VO 

CO 

CM 

00 

CM 

o 

00 

CO 

© 

NT 

CO 

faH 

CM 

vo 

vo 

CM 

p 

VO 

00 

o 

rt 

.s 

’T 

© 

CM 

© 

«“ ‘ 

© 

N- 1 

N-1 

N“ < 

*“N 

*— i 

co 

CM 

CM 

—N 

*— * 

CM 

CM 

7 

CM 

*7 

CM 

o 

*7 

£ 

p 

00 

© 

p 

o 

m 

VO 

vO 

00 

o 

*T 

On 

o 

CM 

NT 

vo 

VO 

CM 

CM 

VO 

o 

Ov 

00 

On 

© 

o 

in 

g. 

CM 

*— « 

CM 

co 

CO 

CM 

CM 

CM 

CO 

CO 

— • 

CM 

CM 

CM 

CO 

CO 

CO 

t 

CM 

*-* 

CM 

CM 

CM 

CM 

CO 

CM 

NT 

CO 

CM 

CO 

e 

© 

m 

NT 

CO 

VO 

CM 

nt 

P 

p 

O 

O 

CM 

OO 

CM 

VO 

vo 

ON 

CM 

CM 

VO 

NT 

p 

On 

OO 

CO 

s 

CO 

CM 

i 

CO 

CO 

co 

CO 

*-« 

CM 

CM 

CM 

CM 

CM 

CM 

*-4 

© 

CM 

CM 

»— i 

CO 

CO 

NT 

CM 

CM 

2 

VO 

CO 

00 

© 

CM 

in 

CM 

nt 

On 

VO 

VO 

CM 

VO 

CO 

CO 

CM 

CM 

O 

CM 

-04 

00 

CM 

NT 

o 

p 

© 

H 

O 

*7 

CM 

CM 

O 

O 

O 

o 

*7 

O 

© 

© 

© 

O 

O 

O 

O 

O 

© 

o 

o 

o 

m 

p 

00 

ON 

CM 

VO 

CO 

VO 

VO 

VO 

00 

vo 

VO 

ON 

o 

CM 

NT 

OV 

CM 

P 

NT 

OO 

NT 

p 

VO 

CO 

<N 

CM 

CM 

O 

CM 

CM 

CM 

o 

CM 

CM 

CO 

CM 

CM 

CM 

CM 

CM 

CO 

CM 

CO 

CM 

CM 

VD 

CM 

CO 

VO 

NO 

CM 

On 

VO 

VO 

T 

m 

p- 

CM 

VO 

© 

CM 

On 

CM 

CM 

ON 

NT 

VO 

NT 

p 

ON 

NO 

P 

ON 

E 

CO 

CM 

CO 

CM 

*-* 

*"* 

CM 

CM 

o 

CM 

CM 

CO 

CM 

CM 

© 

CM 

CM 

CM 

CO 

CM 

CO 

CM 

cm 

o 

3 

-04 

o 

oo 

VO 

OO 

p 

r- 

Nf 

vO 

© 

VO 

P- 

P- 

Nt 

© 

CM 

VO 

t 

NT 

© 

p 

-09 

t 

H 

CM 

CM 

o 

O 

© 

© 

O 

*7 

CM 

CO 

CM 

CM 

CM 

CM 

CM 

CM 

l 

p 

o 

CO 

CO 

OO 

ON 

m 

CM 

CM 

L 

CO 

OO 

© 

p- 

NJ" 

P; 

On 

O 

00 

? 

00 

On 

co 

VO 

P 

CM 

CM 

p 

00 

E-1 

*7 

o 

o 

CM 

<7 

© 

o 

71 

*7 

1 

© 

*7 

CM 

CM 

© 

*T 

*7 

CM 

© 

© 

o 

CO 

CO 

ON 

P 

VO 

CO 

in 

NT 

in 

VO 

NO 

NT 

On 

VO 

On 

00 

O 

00 

On 

CM 

CO 

On 

CO 

CM 

oo 

CM 

m 

in 

l 

ON 

< 

<N 

CM 

1-* 

CM 

CM 

CM 

CM 

CM 

CM 

CM 

wm 

CM 

CO 

CM 

O 

CM 

CM 

CM 

CM 

CO 

CM 

CO 

CM 

Q 

NT 

oo 

p 

VO 

VO 

On 

ON 

CM 

oo 

P- 

CO 

O 

P- 

1 

On 

CO 

On 

CO 

ON 

oo 

ON 

CM 

NT 

NT 

VO 

VO 

vo 

,  , 

NT 

cm 

o 

CO 

CM 

CM 

CM 

CO 

CM 

CM 

w— 

CO 

CM 

CM 

© 

CM 

© 

CM 

CM 

CO 

CM 

CO 

CO 

CM 

SM 

VO 

ON 

CM 

CO 

CO 

NT 

ON 

CO 

CM 

CO 

O 

m 

VO 

o 

OO 

vo 

© 

P 

vo 

NT 

VO 

On 

CO 

oo 

CO 

, 

m 

On 

CM 

CM 

O 

CO 

CO 

o 

»— « 

CM 

CM 

CM 

CM 

CM 

O 

o 

*“ ' 

© 

CM 

CM 

CO 

CM 

H 

NT 

CM 

t 

oo 

VO 

OO 

O 

o 

VO 

o 

o 

OO 

co 

CM 

VO 

CO 

CM 

VO 

CO 

NT 

CO 

00 

VO 

P 

Ov 

O 

On 

m 

, 

U 

CO 

* 

CM 

CM 

CM 

O 

*— i 

CM 

CO 

CO 

«— 1 

CM 

CM 

CM 

CM 

CM 

© 

CM 

CM 

»— < 

CO 

CO 

CO 

CM 

CM 

,  , 

nT 

ON 

ON 

On 

CM 

00 

P 

On 

O 

ON 

VO 

ON 

o 

OO 

ON 

| 

P* 

CM 

CM 

OO 

CM 

VO 

P 

m 

CM 

, 

P 

NT 

w 

CO 

o 

CM 

CM 

CM 

CM 

o 

CM 

CM 

CM 

NT 

CM 

© 

O 

CM 

*“ 1 

1-H 

CM 

CM 

CM 

CM 

CM 

CM 

n 

NT 

m 

00 

VO 

vO 

P 

V-) 

CM 

p 

VO 

CM 

o 

NT 

vO 

vO 

CO 

VO 

VD 

VO 

o 

, 

VO 

, 

OO 

OO 

CO 

P 

OO 

P 

s 

CM 

o 

CM 

CM 

CM 

1”H 

CM 

o 

CM 

o 

CM 

CM 

CM 

Nf 

O 

7 

CM 

CM 

CM 

CM 

CM 

CM 

CO 

CO 

CO 

CM 

CM 

CO 

l 

On 

CO 

vO 

00 

P 

O 

CM 

On 

[ 

O 

vO 

© 

| 

© 

CM 

NT 

VO 

OO 

VO 

© 

On 

Ov 

NT 

NT 

CO 

CO 

2 

-T 

o 

CM 

1 

co 

NT 

CM 

CO 

NT 

CM 

CM 

CM 

CO 

CO 

CM 

CO 

CO 

CM 

o 

CM 

CO 

CO 

CO 

CO 

CM 

CO 

CO 

CM 

On 

vO 

OO 

oo 

© 

oo 

VO 

VO 

P* 

CM 

ON 

p* 

vo 

VO 

© 

CO 

P 

1 

VO 

NT 

NO 

in 

CO 

m 

OO 

CM 

< 

CO 

© 

CM 

CO 

CM 

*“ ' 

© 

O 

O 

CM 

CO 

CM 

© 

CM 

CM 

1-H 

CO 

CM 

CM 

CM 

CM 

CO 

ON 

On 

vO 

P 

oo 

m 

p 

vO 

NO 

ON 

n3" 

P- 

vo 

oo 

NT 

NO 

ON 

CO 

CO 

vO 

NT 

vO 

NT 

CO 

NT 

CO 

»n 

.U 

o 

o 

CM 

CM 

CM 

CM 

o 

o 

o 

CM 

CM 

© 

© 

o 

CM 

NT 

o 

oo 

CM 

vO 

NT 

vn 

VO 

CO 

Ov 

NT 

O 

p- 

On 

VO 

m 

p 

VO 

© 

oo 

CO 

NO 

© 

NT 

On 

CO 

ON 

© 

z 

o 

o 

CM 

CM 

CM 

CM 

O 

o 

o 

O 

© 

CM 

CM 

*— 1 

o 

O 

© 

o 

o 

PC 

NT 

OO 

00 

p 

CM 

o 

vO 

vO 

00 

NO 

vO 

OO 

p- 

CO 

VO 

CO 

p 

VO 

VO 

00 

CM 

OO 

On 

vO 

p 

CO 

co 

O 

CO 

CO 

CM 

O 

CM 

O 

o 

*— 

CO 

*— 1 

© 

CM 

CM 

CM 

CM 

o 

NT 

CM 

00 

On 

NT 

O 

On 

O 

VO 

NT 

CM 

On 

00 

vo 

oo 

On 

CO 

p 

vo 

VO 

■ 

vo 

© 

CM 

in 

CO 

VO 

NT 

CM 

CO 

CO 

' 

CM 

O 

*■“* 

1 

O 

CO 

© 

O 

© 

CM 

CM 

CM 

CM 

CM 

CM 

CO 

CM 

CM 

On 

VO 

CO 

CM 

On 

p 

1 

CO 

CM 

NT 

OO 

vO 

CO 

vo 

CM 

© 

CO 

© 

CO 

p 

oo 

OO 

CM 

P 

VO 

oo 

, 

CO 

CO 

5 

NT 

O 

*— 1 

CO 

CO 

CM 

CO 

CO 

CO 

CO 

CO 

NT 

CM 

CO 

CO 

CO 

*-« 

CM 

CM 

CM 

CM 

NT 

CO 

NT 

NT 

CO 

CO 

p 

NT 

, 

oo 

V-J 

O 

NT 

CM 

NT 

|  , 

o 

NT 

o 

© 

oo 

, 

pN 

vo 

CO 

o 

NO 

NT 

CM 

i 

On 

On 

On 

CO 

p 

m 

O 

CO 

o 

CM 

CO 

CO 

NT 

CM 

O 

CM 

o 

CM 

NT 

CM 

CM 

CM 

CM 

CM 

CM 

CM 

CO 

CM 

Z 

CM 

VO 

VO 

oo 

CO 

NT- 

■NT 

NT 

ON 

CM 

CM 

CM 

P> 

P- 

O 

VO 

VO 

VO 

oo 

© 

© 

© 

P 

, 

CO 

in 

m 

p 

P 

p 

p 

P 

p 

p 

NT 

NT 

CO 

CM 

"T 

On 

On 

VO 

ON 

On 

vo 

oo 

oo 

00 

p 

oo 

CM 

CM 

On 

o 

p* 

P 

CM 

CM 

CM 

CM 

CM 

CM 

VO 

CO 

P- 

P- 

pN 

p 

p 

■  nt 

NT 

CO 

oo 

P 

00 

00 

CM 

CM 

2 

c 

o 

*c 

[T. 

CM 

f  T  . 

g 

> 

< 

oi 

9 

S 

CO 

CM 

fT  ■ 

% 

PU 

% 

H 

Uh 

CO 

X 

CO 

o 

O 

2 

5 

o 

2 

o 

s 

w 

CU 

o 

< 

O 

s 

Q 

< 

2 

U 

< 

o 

c 

u 

2 

E 

o 

o 

o 

— ! 

o 

C 

2 

3 

ep 

Cu. 

o 

< 

U 

£ 

s 

Pu 

2 

D 

CO 

2 

£ 

GfS 

Cu 

n 

tu 

CO 

CQ 

E 

co 

Uh 

s 

Cu 

£ 

2 

CO 

uu 

2 

CO 

PU 

X 

X 

2  2 

£ 

2 

Cu 

CM 

§ 

-g 

C 

0 

CO 

U 

s 

CM 

w 

1 

S 

o 

CM 

o 

> 

CM 

> 

CO 

> 

NT 

> 

i 

CM 

2 

z 

CM 

H 

p 

o 

s 

2 

CM 

2 

CO 

2 

2 

5 

CM 

55 

CO 

CO 

s 

CM 

2 

CO 

< 

< 

< 

< 

< 

< 

< 

< 

< 

< 

< 

< 

< 

w 

w 

W 

CZ3 

ta 

PU 

uu 

O 

o 

o 

2 

o 

O 

O 

& 

1-9 


ECAT  Test  Measures  Used  as  Predictors 

CT  =  Mental  Counters  Proportion  Correct  SM  =  Sequential  Memory  Proportion  Correct  ID  =  Integrating  Details  Proportion  Correct 

AO  =  Assembling  Objects  Proportion  Correct  T1  =  l -Hand  Tracking  Mean  1000*log(l  +  RMS  (Attempted))  T2  =  2-Hand  Tracking  Mean  1000*log(l  +  RMS  (Attempted)) 

FR  =  Figural  Reasoning  Proportion  Correct  SO  =  Spatial  Orientation  Proportion  Correct  TI  =  Target  Identification  Mean  Clipped  Decision  RTs 


Range-Corrected  Simple  Validities  for  all  Army  Criteria 
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Mental  Counters  Proportion  Correct  SM  =  Sequential  Memory  Proportion  Correct  ID  =  Integrating  Details  Proportion  Correct 

Assembling  Objects  Proportion  Correct  T1  -  1-Hand  Tracking  Mean  1000*log(l  +  RMS  (Attempted))  T2  =  2-Hand  Tracking  Mean  1000*log(l  +  RMS(Attempted)) 

Figural  Reasoning  Proportion  Correct  SO  =  Spatial  Orientation  Proportion  Correct  TI  =  Target  Identification  Mean  Clipped  Decision  RTs 


Range-Corrected  Simple  Validities  for  all  Air  Force  Criteria 
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Mental  Counters  Proportion  Correct  SM  =  Sequential  Memoiy  Proportion  Correct  ID  =  Integrating  Details  Proportion  Correct 

Assembling  Objects  Proportion  Correct  T1  =  i-Hand  Tracking  Mean  1000*log(l  +  RMS  (Attempted))  T2  =  2-Hand  Tracking  Mean  1000*log(l  +  RMS(Attempted)) 

Figure!  Reasoning  Proportion  Correct  SO  =  Spatial  Orientation  Proportion  Correct  TI  =  Target  Identification  Mean  Clipped  Decision  RTs 
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Appendix  J 

Factor  Validities  and  Incremental  Validities  for  All  Criteria 


J-0 


Table  J-l 


ECAT  Factor  Incremental  Validities  for  all  Army  Criteria 


Uncorrected  Multiple  R _  Corrected  Multiple  R 


School 

Criterion 

Sample 

Size 

ASVAB 

ASVAB 

+ECAT 

Percent 

Variance 

Probability 

ASVAB 

Increase 

Percent 

11H(A)1 

TOALL 

554 

.204 

.222 

0.792 

2.299x10"' 

.247 

.006 

2.3 

11H(A)2 

EVT1TO 

556 

216 

.327 

3.427 

3.499x10"* 

.383 

.034 

8.8  ** 

11H(A)3 

EVT2TO 

555 

.190 

.235 

2.054 

1.104xl0"2 

.247 

.033 

13.4* 

11H(A)4 

EVT3TO 

550 

.260 

.300 

2.466 

4.192xl0"3 

.349 

.026 

7.3  ** 

11H(A)5 

EVTSUM 

546 

.279 

.334 

3.830 

1.513x10"* 

.370 

.038 

10.4  ** 

11H(A)6 

TO_l 

542 

.169 

.221 

2.137 

1.019xl0~2 

.219 

.050 

22.8* 

11H(B)1 

TOALL 

320 

.244 

.266 

1.199 

2.929x10"' 

.344 

.000 

0.0 

11H(B)2 

EVT1TO 

320 

.244 

.366 

8.614 

1.039xl0"5 

.304 

.089 

29.3  ** 

11H(B)3 

EVT2TO 

319 

.292 

.407 

9.656 

2.575x10"* 

.401 

.082 

20.5  ** 

11H(B)4 

EVT3TO 

319 

.191 

.248 

2.686 

4.099xl0"2 

.238 

.036 

15.2* 

11H(B)5 

EVTSUM 

316 

.292 

.412 

10.186 

1.432x10"* 

.381 

.084 

21.9  ** 

11H(B)6 

TO_l 

319 

.102 

.281 

7.457 

5.377xl0"s 

.099 

.194 

196.5  ** 

1 1H(B)7 

TO_2 

320 

.119 

.278  • 

6.830 

1.247xlOJ 

.106 

.165 

156.2  ** 

11H(B)8 

TO_3 

319 

.099 

.266 

6.536 

1.938x10"* 

.052 

.196 

376.2  ** 

11H(B)9 

ITVTOW 

318 

.087 

.303 

9.303 

4.363x10"* 

.088 

.214 

243.6  ** 

13F1 

FSG 

821 

.523 

.583 

10.158 

5.551xl0"'7 

.778 

.029 

3.7  ** 

13F2 

MPRAD 

821 

.503 

.586 

13.706 

<  10"'7 

.806 

.045 

5.6  ** 

13F3 

FIRING 

821 

.419 

.444 

2.771 

5.830xl0's 

.713 

.012 

1.7  ** 

19K1 

COMM 

1158 

.037 

.044 

0.056 

8.858x10"' 

.000 

.000 

, 

19K2 

WEAPON 

1325 

.163 

.165 

0.094 

7.434x10"' 

.188 

.000 

0.0 

19K3 

LANDNAV  1192 

.162 

.171 

0.319 

2.870x10"' 

.188 

.005 

2.5 

19K4 

LOADER 

1313 

.048 

.050 

0.020 

9.664x10"' 

.029 

.000 

0.0 

19K5 

MAINT 

1329 

.087 

.090 

0.055 

8.657x10"' 

.089 

.000 

0.0 

19K6 

NBC 

1313 

.111 

.124 

0.330 

2.305x10"' 

.137 

.005 

4.0 

19K7 

AVERAG 

1106 

.193 

.196 

0.159 

6.258x10"' 

.390 

.000 

0.0 

Table  J-2 


ECAT  Factor  Incremental  Validities  for  all  Air  Force  Criteria 


Uncorrected  Multiple  R _  Corrected  Multiple  R 


School 

Criterion 

Sample 

Size 

ASVAB 

ASVAB 

+ECAT 

Percent 

Variance 

Probability 

°f 

ASVAB 

Increase 

Percent 

APSl 

FSG 

446 

.526 

.559 

5.326 

4.546xl0'5 

.823 

.013 

1.6  ** 

APS2 

ZHRS 

446 

.391 

.449 

6.189 

8.122x10-* 

.669 

.030 

4.5  ** 

APS3 

AFPT70 

432 

.265 

.391 

9.807 

1.225x10-* 

.373 

.102 

27.3  ** 

ATC1 

FSG 

484 

.387 

.417 

2.848 

3.884  xl0‘3 

.722 

.015 

2.0  ** 

ATC2 

BLK2 

349 

.352 

.373 

1.739 

1.172x10-' 

.646 

.009 

1.3 

ATC3 

BLK3A 

529 

.125 

.189 

2.086 

1. 305x10"* 

.266 

.077 

28.7  * 

ATC4 

BLK3B 

217 

.046 

.210 

4.401 

2.899x1 O'2 

.000 

.252 

.  * 

ATC5 

BLK5A 

500 

.215 

.327 

6.809 

4.224xl0"7 

.460 

.103 

22.4  ** 

ATC6 

BLK5B 

495 

.182 

.217 

1.444 

7.225xl0"2 

.430 

.028 

6.5 

ATC7 

FAA 

536 

.480 

.503 

3.065 

1.159xl0’3 

.751 

.011 

.1.5  ** 

ATC(A)1 

FSG 

200 

.355 

.430 

7.201 

3.874xl0"3 

.675 

.036 

5.3  ** 

ATC(A)2 

BLK3A 

221 

.162 

.188 

0.935 

5.752x10-’ 

.169 

.027 

16.1 

ATC(A)3 

BLK3B 

217 

.046 

.210 

4.401 

2.899x1 0"2 

.000 

.252 

.  * 

ATC(A)4 

BLK5A 

205 

.153 

.299 

7.260 

3.130xl0"3 

.479 

.121 

25.2  ** 

ATC(A)5 

BLK5B 

204 

.161 

.191 

1.087 

5.468x10-’ 

.295 

.015 

5.1 

ATC(A)6 

FAA 

251 

.492 

.524 

4.488 

1.352x10* 

.740 

.013 

1.7* 

ATC(B)1 

FSG 

284 

.418 

.426 

0.888 

4.852x10-’ 

.749 

.003 

0.4 

ATC(B)2 

BLK2 

349 

.352 

.373 

1.739 

1.172x10-’ 

.646 

.009 

1.3 

ATC(B)3 

BLK3A 

308 

.138 

.237 

3.929 

8.962 xlO"3 

.198 

.160 

80.9  ** 

ATC(B)4 

BLK5A 

295 

.272 

.372 

7.488 

1.181X10-* 

.454 

.092 

20.2  ** 

ATC(B)5 

BLK5B 

291 

.226 

.274 

2.594 

6.405X10-2 

.528 

.035 

6.7 

ATC(B)6 

FAA 

285 

.459 

.474 

1.801 

1.752x10-’ 

.717 

.011 

1.5 

Table  J-3 


ECAT  Factor  Incremental  Validities  for  all  Navy  Criteria 


Uncorrected  Multiple  R _  Corrected  Multiple  R 


School 

Criterion 

Sample 

Size 

ASVAB 

ASVAB 

+ECAT 

Percent 

Variance 

Probability 
°f  ^3,A?-8 

ASVAB 

Increase 

Percent 

ACl 

FSG 

72 

.569 

.584 

2.559 

6.527x10-' 

.820 

.000 

0.0 

AC2 

PERF 

76 

.228 

.383 

11.095 

6.561x10" 

.399 

.146 

36.7 

AC3 

FAA 

76 

.339 

.411 

6.465 

2.317x10-' 

.523 

.054 

10.2 

AE1 

FSG 

278 

.475 

.530 

7.702 

1. 642x10" 

.666 

.025 

3.8  ** 

AE2 

SUM2 

273 

.387 

.453 

7.059 

4.178x10" 

.596 

.025 

4.2  ** 

AMS1 

FSG 

244 

.581 

.583 

0.209 

9.203x10" 

.846 

.000 

0.0 

AMS2 

PERF 

244 

.355 

.383 

2.356 

1.382x10" 

.639 

.011 

1.7 

AOl 

FSG 

234 

.488 

.499 

1.428 

3.601x10" 

.716 

.006 

0.8 

A02 

PRACTL 

229 

.308 

.328 

1.423 

3.720x10" 

.492 

.011 

2.2 

AVI 

FSG 

544 

.494 

.512 

2.505 

4.082x10" 

.804 

.005 

0.6  ** 

AV2 

BSCAV 

192 

.493 

.517 

3.300 

1.121x10" 

.839 

.004 

0.5 

AV3 

ADVAV 

192 

.316 

.326 

0.703 

7.311x10" 

.690 

.000 

0.0 

AV4 

PERFORM 

352 

.345 

.373 

2.342 

4.653x10" 

.656 

.015 

2.3  * 

EMI 

FSG 

797 

.437 

.439 

0.214 

6.392x10" 

.684 

.000 

0.0 

EM2 

PHASE  1 

797 

.451 

.455 

0.387 

3.844x10" 

.723 

.000 

0.0 

EN1 

FSG 

750 

.581 

.585 

0.600 

2.174x10" 

.764 

.001 

0.1 

ET1 

FSG 

86 

.439 

.520 

10.548 

4.874x10" 

.782 

.040 

5.1  * 

ET2 

FSG2 

86 

.465 

.516 

6.854 

1.575x10" 

.811 

.029 

3.6 

ET3 

PERF 

86 

.384 

.434 

5.084 

2.733x10" 

.676 

.041 

6.1 

FC1 

FSG 

778 

.475 

.490 

1.934 

2.032x10" 

.821 

.005 

0.6  ** 

FC2 

RADAR 

780 

.339 

.368 

2.344 

4.609x10" 

.734 

.012 

1.6  ** 

GM1 

FSG 

420 

.402 

.415 

1.236 

1.669x10" 

.727 

.001 

0.2 

GM2 

HALF1 

420 

.428 

.448 

2.256 

2.671x10" 

.761 

.004 

0.5* 

GM3 

HALF2 

397 

.424 

.428 

0.441 

6.340x10" 

.726 

.000 

0.0 

MM1 

FSG 

801 

.391 

.402 

1.084 

3.583x10" 

.555 

.005 

0.9* 

OS1 

FSG 

713 

.549 

.567 

2.973 

1.234x10" 

.798 

.009 

2  1  ** 

OS2 

WRIT 

815 

.474 

.481 

0.864 

7.373x10" 

.756 

.003 

0.4 

OS3 

PERF 

815 

.504 

.548 

6.752 

2.060x10"’ 

.782 

.027 

3.5  ** 

RM1 

FSG 

277 

.508 

.533 

3.646 

2.178x10" 

.767 

.011 

1.4* 

RM2 

PHASE3 

277 

.367 

.388 

1.863 

1.736x10" 

.677 

.004 

0.6 

*  p  <  .05. 


**  p  <  .01. 


Table  J-4 


Incremental  Validities  from  Adding  one  ECAT  Factor  to  Four  ASVAB  Factors 
All  Significant  Criteria  from  Full  Model 


School 

Criterion 

Memory 

Psychomotor 

Space 

11H(A)2 

EVT1TO 

.010* 

.026** 

.028** 

1 1H(A)3 

EVT2TO 

.008 

.033** 

.022* 

11H(A)4 

EVT3TO 

.010* 

.025** 

.016** 

1 1H(A)5 

EVTSUM 

.012* 

.034** 

.027** 

11H(A)6 

TO_l 

.000 

.055** 

.003 

11H(B)2 

EVT1TO 

029** 

.090** 

.036** 

1 1H(B)3 

EVT2TO 

.035** 

.076** 

.032** 

11H(B)4 

EVT3TO 

.000 

.032* 

.000 

1 1H(B)5 

EVTSUM 

.021** 

.086** 

.023** 

1 1H(B)6 

TO_l 

.063 

.098** 

.000 

1 1H(B)7 

TO_2 

.000 

.167** 

.049** 

11H(B)8 

TO_3 

.000 

.205** 

.063* 

11H(B)9 

ITVTOW 

.000 

.178** 

.039** 

13F1 

FSG 

.018** 

.007** 

.028** 

13F2 

MPRAD 

024** 

.006** 

.045** 

13F3 

FIRING 

.011** 

.005** 

.009** 

APS1 

FSG 

.009** 

.000 

.006** 

APS2 

ZHRS 

.023** 

.000 

.000 

APS3 

AFPT70 

.051** 

.015* 

.034** 

ATC1 

FSG 

.012** 

.006* 

.013** 

ATC3 

BLK3A 

.050* 

.060** 

.021 

ATC4 

BLK3B 

.270** 

.000 

.122 

ATC5 

BLK5A 

.067** 

.056** 

.088** 

ATC7 

FAA 

.008** 

.003 

.012** 

ATC(A)1 

FSG 

.031** 

.021** 

.018** 

ATC(A)3 

BLK3B 

.270** 

.000 

.122 

ATC(A)4 

BLK5A 

.089* 

.047* 

.120** 

ATC(A)6 

FAA 

.011** 

.005* 

.012** 

ATC(B)3 

BLK3A 

.134* 

.089* 

.030 

ATC(B)4 

BLK5A 

.060** 

.053** 

.078** 

AC2 

PERF 

.150* 

.019 

.142 

AE1 

FSG 

.024** 

.003* 

.022** 

AE2 

SUM2 

.024** 

.000 

.013** 

AVI 

FSG 

.005** 

.001 

.004** 

AV4 

PERFORM 

.009 

.014* 

.011* 

ET1 

FSG 

.024 

.022* 

.038* 

FC1 

FSG 

.000 

.000 

.003** 

FC2 

RADAR 

.002* 

.004 

.000 

GM2 

HALF1 

.000 

.001 

.005** 

MM1 

FSG 

.000 

.000 

.006** 

OS1 

FSG 

.007** 

.000 

.008** 

OS3 

PERF 

.020** 

.008** 

.025** 

RM1 

FSG 

.005* 

.001 

.004 

*  p  <  .05.  **p<.01. 
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Table  J-5 


Uncorrected  Factor  Validities  for  all  Army  Criteria 


ASVAB  Factor _  _ ECAT  Factor 


School 

Criterion 

N 

Verbal 

Math 

Tech 

Clerical 

Space 

Memory 

Psychomoto 

r 

11H(A)1 

TOALL 

554 

.094 

.167 

.149 

.105 

.173 

.165 

-.058 

1 1H(A)2 

EVT1TO 

556 

.202 

.199 

.258 

.048 

.268 

.180 

-.224 

11H(A)3 

EVT2TO 

555 

.126 

.156 

.162 

.075 

.197 

.146 

-.183 

11H(A)4 

EVT3TO 

550 

.227 

.202 

.219 

.054 

.234 

.177 

-.209 

11H(A)5 

EVTSUM 

546 

.214 

.210 

.256 

.060 

.274 

.194 

-.242 

11H(A)6 

TO_l 

542 

.128 

.151 

.086 

.132 

.138 

.107 

-.188 

11H(B)1 

TOALL 

320 

.229 

.104 

.142 

.106 

.155 

.146 

-.111 

11H(B)2 

EVT1TO 

320 

.211 

.188 

.201 

.115 

.278 

.237 

-.326 

11H(B)3 

EVT2TO 

319 

.250 

.194 

.178 

.205 

.278 

.262 

-.335 

11H(B)4 

EVT3TO 

319 

.103 

.138 

.161 

.082 

.103 

.067 

-.181 

1 1H(B)5 

EVTSUM 

316 

.244 

.221 

.224 

.174 

.288 

.251 

-.361 

11H(B)6 

TO_l 

319 

.002 

.033 

.028 

-.058 

.091 

-.061 

-.165 

11H(B)7 

TO_2 

320 

.000 

-.045 

.058 

-.104 

.099 

.019 

-.230 

11H(B)8 

TO_3 

319 

.058 

.043 

.050 

.076 

.143 

.076 

-.250 

11H(B)9 

ITVTOW 

318 

.008 

.004 

.047 

-.067 

.125 

-.008 

-.244 

13F1 

FSG 

821 

.388 

.434 

.379 

.263 

.496 

.401 

-.308 

13F2 

MPRAD 

821 

.369 

.430 

.368 

.234 

.525 

.407 

-.294 

13F3 

FIRING 

821 

.318 

.353 

.292 

.217 

.329 

.298 

-.236 

19K1 

COMM 

1158 

.010 

.004 

-.019 

.025 

-.017 

-.012 

.013 

19K2 

WEAPON 

1325 

.097 

.153 

.098 

.091 

.103 

.102 

-.078 

19K3 

LANDNAV 

1192 

.063 

.147 

.079 

.100 

.103 

.109 

-.098 

19K4 

LOADER 

1313 

.038 

.014 

.039 

.012 

.022 

.019 

-.026 

19K5 

MAINT 

1329 

.043 

.082 

.053 

.030 

.048 

.050 

-.035 

19K6 

NBC 

1313 

.068 

.086 

.046 

.093 

.043 

.037 

-.076 

19K7 

AVERAGE 

1106 

.106 

.169 

.101 

.139 

.135 

.126 

-.106 
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Table  J-6 


Uncorrected  Factor  Validities  for  all  Air  Force  Criteria 


ASVAB  Factor _  _ ECAT  Factor 


School 

Criterion 

N 

Verbal 

Math 

Tech 

Clerical 

Space 

Memory 

Psychomotor 

APS1 

FSG 

446 

.361 

.472 

.316 

.163 

.341 

.329 

-.095 

APS2 

ZHRS 

446 

-.239 

-.327 

-.102 

-.244 

-.176 

-.306 

.025 

APS3 

AFPT70 

432 

.113 

.090 

-.097 

.206 

.149 

.226 

.131 

ATC1 

FSG 

484 

.311 

.313 

.301 

.072 

.320 

.223 

-.230 

ATC2 

BLK2 

349 

.234 

.289 

.297 

-.001 

.265 

.172 

-.246 

ATC3 

BLK3A 

529 

.083 

.123 

.056 

.028 

.110 

.138 

-.138 

ATC4 

BLK3B 

217 

-.044 

-.029 

-.018 

-.001 

.076 

.178 

-.030 

ATC5 

BLK5A 

500 

.044 

.197 

.086 

.067 

.287 

.231 

-.233 

ATC6 

BLK5B 

495 

.110 

.140 

.022 

.145 

.115 

.144 

-.101 

ATC7 

FAA 

536 

.401 

.397 

.349 

.085 

.380 

.246 

-.223 

ATC(A)1 

FSG 

200 

.242 

.311 

.251 

.120 

.340 

.292 

-.305 

ATC(A)2 

BLK3A 

221 

.117 

.129 

-.006 

.064 

.102 

.086 

-.090 

ATC(A)3 

BLK3B 

217 

-.044 

-.029 

-.018 

-.001 

.076 

.178 

-.030 

ATC(A)4 

BLK5A 

205 

.021 

.102 

.109 

.023 

.282 

.185 

-.199 

ATC(A)5 

BLK5B 

204 

.029 

.089 

-.038 

.148 

.101 

.082 

-.074 

ATC(A)6 

FAA 

251 

.355 

.420 

.384 

.049 

.405 

.295 

-.298 

ATC(B)1 

FSG 

284 

.350 

.303 

.340 

.006 

.286 

.160 

-.181 

ATC(B)2 

BLK2 

349 

.234 

.289 

.297 

-.001 

.265 

.172 

-.246 

ATC(B)3 

BLK3A 

308 

.058 

.119 

.105 

-.000 

.120 

.182 

-.175 

ATC(B)4 

BLK5A 

295 

.061 

.260 

.073 

.099 

.297 

.266 

-.254 

ATC(B)5 

BLK5B 

291 

.171 

.177 

.065 

.143 

.126 

.193 

-.121 

ATC(B)6 

FAA 

285 

.391 

.347 

.342 

-.030 

.257 

.182 

-.236 
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Table  J-7 


Uncorrected  Factor  Validities  for  all  Navy  Criteria 


ASVAB  Factor _  _ ECAT  Factor 


School 

Criterion 

N 

Verbal 

Math 

Tech 

Clerical 

Space 

Memory 

Psychomotor 

AC1 

FSG 

72 

.340 

.538 

.381 

.212 

.344 

.342 

-.192 

AC2 

PERF 

76 

-.051 

.054 

-.045 

.209 

.205 

.285 

-.063 

AC3 

FAA 

76 

.242 

.261 

.295 

.118 

.248 

.105 

-.049 

AE1 

FSG 

278 

.385 

.354 

.412 

.130 

.418 

.373 

-.254 

AE2 

SUM2 

273 

.244 

.302 

.346 

.069 

.336 

.334 

-.139 

AMS1 

FSG 

244 

.450 

.441 

.364 

.323 

.239 

.148 

-.085 

AMS2 

PERF 

244 

.155 

.209 

.185 

.265 

.233 

.156 

-.106 

AOl 

FSG 

234 

.262 

.439 

.245 

.308 

.284 

.154 

-.132 

A02 

PRACTL 

229 

.114 

.225 

.076 

.282 

.207 

.168 

-.093 

AVI 

FSG 

544 

.174 

.450 

.235 

.199 

.349 

.285 

-.180 

AV2 

BSCAV 

192 

.249 

.458 

.156 

.110 

.374 

.287 

-.157 

AV3 

ADVAV 

192 

.204 

.285 

.056 

.162 

.170 

.118 

-.132 

AV4 

PERFORM 

352 

.015 

.228 

.243 

.044 

.235 

.181 

-.186 

EMI 

FSG 

797 

.248 

.388 

.302 

.167 

.264 

.208 

-.126 

EM2 

PHASE1 

797 

.236 

'  .422 

.283 

.145 

.271 

.211 

-.110 

EN1 

FSG 

750 

.434 

.443 

.500 

.170 

.393 

.268 

-.211 

ET1 

FSG 

86 

.160 

.284 

.291 

.092 

.412 

.260 

-.349 

ET2 

FSG2 

86 

.166 

.389 

.113 

.291 

.355 

.305 

-.228 

ET3 

PERF 

86 

.026 

.314 

-.078 

.323 

.188 

.232 

-.198 

FC1 

FSG 

778 

.260 

.362 

.277 

.211 

.273 

.120 

-.154 

FC2 

RADAR 

780 

.204 

.148 

.264 

.132 

.138 

-.011 

-.133 

GM1 

FSG 

420 

.272 

.271 

.244 

.142 

.279 

.204 

-.220 

GM2 

HALF1 

420 

.247 

.344 

.203 

.189 

.327 

.233 

-.234 

GM3 

HALF2 

397 

.280 

.187 

.339 

.111 

.242 

.140 

-.168 

MM1 

FSG 

801 

.266 

.288 

.335 

.153 

.295 

.182 

-.159 

OS1 

FSG 

713 

.305 

.507 

.368 

.212 

.453 

.375 

-.252 

OS2 

WRIT 

815 

.311 

.437 

.335 

.127 

.353 

.301 

-.199 

OS3 

PERF 

815 

.212 

.448 

.341 

.208 

.466 

.397 

-.294 

RM1 

FSG 

277 

.421 

.445 

.324 

.174 

.334 

.290 

-.099 

RM2 

PHASE3 

277 

.299 

.330 

.257 

.069 

.281 

.231 

-.114 
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Table  J-8 


Range-Corrected  Factor  Validities  for  all  Army  Criteria 


School 

Criterion 

N 

ASVAB  Factor 

EC  AT  Factor 

Verbal 

Math 

Tech 

Clerical 

Space 

Memory 

Psychomotor 

11H(A)1 

TOALL 

554 

.158 

.222 

.200 

.142 

.220 

.198 

-.098 

1 1H(A)2 

EVT1TO 

556 

.291 

.281 

.360 

.109 

.356 

.257 

-.297 

11H(A)3 

EVT2TO 

555 

.194 

.203 

.236 

.102 

.254 

.191 

-.236 

11H(A)4 

EVT3TO 

550 

.307 

.283 

.300 

.117 

.313 

.254 

-.268 

11H(A)5 

EVTSUM 

546 

.303 

.290 

.352 

.119 

.357 

.268 

-.313 

11H(A)6 

TO_l 

542 

.180 

.195 

.156 

.179 

.190 

.151 

-.242 

11H(B)1 

TOALL 

320 

.321 

.226 

.233 

.224 

.244 

.231 

-.170 

11H(B)2 

EVT1TO 

320 

.260 

.255 

.281 

.168 

.326 

.282 

-.363 

11H(B)3 

EVT2TO 

319 

.348 

.312 

.282 

.311 

.365 

.354 

-.394 

11H(B)4 

EVT3TO 

319 

.149 

.194 

.217 

.140 

.163 

.112 

-.236 

11H(B)5 

EVTSUM 

316 

.323 

.319 

.323 

.261 

.365 

.325 

-.417 

11H(B)6 

TO_l 

319 

-.024 

.005 

.035 

-.103 

.038 

-.107 

-.147 

11H(B)7 

TO_2 

320 

-.018 

-.043 

.072 

-.112 

.074 

-.010 

-.213 

11H(B)8 

TO_3 

319 

.074 

.070 

.101 

.074 

.141 

.083 

-.258 

11H(B)9 

ITVTOW 

318 

-.021 

-.013 

.060 

-.104 

.077 

-.051 

-.222 

13F1 

FSG 

821 

.586 

.638 

.543 

All 

.638 

.548 

-.409 

13F2 

MPRAD 

821 

.562 

.627 

.531 

.450 

.654 

.548 

-.397 

13F3 

FIRING 

821 

.498 

.537 

.443 

.405 

.489 

.445 

-.334 

19K1 

COMM 

1158 

.017 

.008 

-.011 

.026 

-.014 

-.007 

.020 

19K2 

WEAPON 

1325 

.136 

.185 

.140 

.135 

.138 

.134 

-.105 

19K3 

LANDNAV 

1192 

.101 

.  .178 

.124 

.133 

.140 

.134 

-.136 

19K4 

LOADER 

1313 

.053 

.037 

.054 

.032 

.041 

.036 

-.036 

19K5 

MAINT 

1329 

.071 

.101 

.075 

.061 

.069 

.072 

-.052 

19K6 

NBC 

1313 

.106 

.125 

.084 

.132 

.080 

.070 

-.097 

19K7 

AVERAGE 

1106 

.169 

.224 

.161 

.200 

.188 

.172 

-.147 
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Table  J-9 


Range-Corrected  Factor  Validities  for  all  Air  Force  Criteria 


ASVAB  Factor  _  _ ECAT  Factor 


School 

Criterion 

N 

Verbal 

Math 

Tech 

Clerical 

Space 

Memory 

Psychomotor 

APS1 

FSG 

446 

.681 

.730 

.559 

.495 

.626 

.575 

-.320 

APS2 

ZHRS 

446 

-.534 

-.601 

-.349 

-.503 

-.461 

-.531 

.235 

APS3 

AFPT70 

432 

.268 

.305 

.093 

.337 

.303 

.359 

-.014 

ATC1 

FSG 

484 

.619 

.584 

.531 

.383 

.561 

.487 

-.381 

ATC2 

BLK2 

349 

.511 

.492 

.498 

.255 

.472 

.395 

-.350 

ATC3 

BLK3A 

529 

.174 

.200 

.113 

.141 

.199 

.227 

-.203 

ATC4 

BLK3B 

217 

.075 

.099 

.055 

.106 

.168 

.266 

-.123 

ATC5 

BLK5A 

500 

.247 

.379 

.234 

.281 

.443 

.414 

-.341 

ATC6 

BLK5B 

495 

.292 

.318 

.164 

.306 

.280 

.311 

-.226 

ATC7 

FAA 

536 

.707 

.667 

.588 

.436 

.626 

.533 

-.399 

ATC(A)1 

FSG 

200 

.534 

.581 

.462 

.430 

.556 

.538 

-.426 

ATC(A)2 

BLK3A 

221 

.103 

.159 

.016 

.136 

.132 

.140 

-.163 

ATC(A)3 

BLK3B 

217 

.075 

.099 

.055 

.106 

.168 

.266 

-.123 

ATC(A)4 

BLK5A 

205 

.239 

.323 

.279 

.250 

.434 

.391 

-.317 

ATC(A)5 

BLK5B 

204 

.146 

.213 

.061 

.258 

.204 

.214 

-.193 

ATC(A)6 

FAA 

251 

.675 

.672 

.601 

.415 

.637 

.549 

-.423 

ATC(B)1 

FSG 

284 

.666 

.568 

.581 

.324 

.538 

.447 

-.351 

ATC(B)2 

BLK2 

349 

.511 

.492 

.498 

.255 

.472 

.395 

-.350 

ATC(B)3 

BLK3A 

308 

.122 

.158 

.135 

.060 

.187 

.239 

-.203 

ATC(B)4 

BLK5A 

295 

.246 

.415 

.197 

.313 

.456 

.439 

-.349 

ATC(B)5 

BLK5B 

291 

.415 

.405 

.260 

.338 

.364 

.405 

-.270 

ATC(B)6 

FAA 

285 

.686 

.616 

.577 

.344 

.561 

.501 

-.407 
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Table  J-10 


Range-Corrected  Factor  Validities  for  all  Navy  Criteria 


ASVAB  Factor _  _ ECAT  Factor 


School 

Criterion 

N 

Verbal 

Math 

Tech 

Clerical 

Space 

Memory 

Psychomotor 

AC1 

FSG 

72 

.611 

.805 

.511 

.570 

.620 

.569 

-.309 

AC2 

PERF 

76 

.128 

.319 

.030 

.328 

.391 

.451 

-.227 

AC3 

FAA 

76 

.490 

.510 

.438 

.367 

.414 

.250 

-.075 

AE1 

FSG 

278 

.564 

.563 

.556 

.364 

.587 

.516 

-.371 

AE2 

SUM2 

273 

.423 

.497 

.479 

.286 

.497 

.453 

-.256 

AMS1 

FSG 

244 

.708 

.707 

.705 

.480 

.618 

.507 

-.403 

AMS2 

PERF 

244 

.401 

.451 

.483 

.373 

.485 

.397 

-.328 

AOl 

FSG 

234 

.557 

.657 

.488 

All 

.551 

.428 

-.387 

A02 

PRACTL 

229 

.339 

.391 

.225 

.401 

.373 

.338 

-.246 

AVI 

FSG 

544 

.620 

.763 

.600 

.470 

.652 

.568 

-.405 

AV2 

BSCAV 

192 

.693 

.789 

.616 

.453 

.680 

.580 

-.395 

AV3 

ADVAV 

192 

.600 

.618 

.451 

.436 

.484 

.411 

-.347 

AV4 

PERFORM 

352 

.316 

.443 

.478 

.178 

.457 

.361 

-.352 

EMI 

FSG 

797 

.532 

.641 

.503 

.415 

.524 

.439 

-.311 

EM2 

PHASE  1 

797 

.531 

.668 

.493 

.407 

.539 

.450 

-.307 

EN1 

FSG 

750 

.610 

.628 

.655 

.374 

.569 

.430 

-.372 

ET1 

FSG 

86 

.615 

.699 

.642 

.444 

.734 

.616 

-.532 

ET2 

FSG2 

86 

.651 

.767 

.577 

.592 

.738 

.661 

-.506 

ET3 

PERF 

86 

.489 

.631 

.394 

.526 

.574 

.519 

-.506 

FC1 

FSG 

778 

.688 

.750 

.653 

.478 

.651 

.495 

-.400 

FC2 

RADAR 

780 

.573 

.539 

.583 

.357 

.491 

.324 

-.371 

GM1 

FSG 

420 

.625 

.651 

.580 

.413 

.573 

.466 

-.379 

GM2 

HALF1 

420 

.621 

.697 

.568 

.452 

.610 

.493 

-.382 

GM3 

HALF2 

397 

.611 

.597 

.615 

.379 

.542 

.419 

-.338 

MM1 

FSG 

801 

.417 

.459 

.465 

.294 

.444 

.332 

-.279 

OS1 

FSG 

713 

.622 

.734 

.507 

.565 

.629 

.567 

-.369 

OS2 

WRIT 

815 

.589 

.664 

.484 

.473 

.546 

.494 

-.334 

OS3 

PERF 

815 

.523 

.669 

.479 

.522 

.631 

.571 

-.409 

RM1 

FSG 

277 

.652 

.663 

.504 

.509 

.565 

.518 

-.271 

RM2 

PHASE3 

277 

.509 

.534 

.431 

.356 

.480 

.413 

-.255 
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Appendix  K 

SAS  Program  for  Stepwise  Meta-Analysis 


K-0 


METASTEP.SAS 


OPTIONS  LS=7  9  MPRINT  DQUOTE  ; 

/*  PROGRAMS  TO  DO  STEPWISE  META-ANALYSIS; 

LOW-LEVEL  ROUTINES  APPEAR  FIRST.  */ 

% MACRO  NUMLIST (LIST) ;  /*  RETURNS  THE  NUMBER  OF  ELEMENTS  OF  LIST  */ 

% LOCAL  I; 

%LET  1=1; 

%DO  % WHILE (  %SCAN (&LIST, &I)  NE)  ; 

%LET  I  =  %EVAL (& I  +  1) ; 

%END; 

%EVAL (&I  -  1) 

%MEND  NUMLIST; 

%MACRO  REMOVE (J, LIST, N) ;  /*  REMOVES  THE  JTH  MEMBER  OF  THE  LIST  */ 

% LOCAL  M;  /*  N  =  NUMBER  OF  ELEMENTS  OF  LIST  */ 

%DO  M=  1  %TO  &N; 

%IF  &M  NE  &J  %THEN  %SCAN (&LIST, &M) ; 

%END  ; 

%MEND  REMOVE; 

% MACRO  REMOVAL (OUTS, LIST) ;  /*  RETURNS  LIST  -  OUTS  */ 

% LOCAL  M  N  V; 

%LET  N  =  %NUMLIST (&LIST) ; 

%DO  M=  1  %TO  &N; 

%LET  V  =  %SCAN ( &LIST, &M) ; 

%IF  % INDEX (SOUTS,  &V)  =  0  %THEN  &V; 

%END; 

' %MEND  REMOVAL; 

% MACRO  STEP (RATING) ;  /*  FOR  A  GIVEN  SCHOOL,  COMPUTES  THE  MULTIPLE  R 

FROM  ADDING  OR  DELETING  EACH  PREDICTOR  */ 

%IF  &NVL2  NE  0  %THEN  %DO;  /*  SKIP  IF  NO  MORE  VARIABLES  */ 

DATA  TEMP ; SET  LAWCOR.N&RATING; IF  GROUP=3;  /*  GET  LAWLEY-CORRECTED  */ 
DATA  _NULL_;  SET  TEMP  END=LAST;  /*  CORRELATIONS  FROM  DISK*/ 

IF  LAST  THEN  CALL  SYMPUT ( ’ DEP '  ,  _NAME_) ; 

DATA  _NULL_;  SET  ALL . RELX;  /*  RETRIEVE  RELIABILITY  FROM  DISK  */ 

IF  INDEX (SCHOOL, TRIM (LEFT ("&RATING") ) )  >  0  THEN  DO; 

REL  =  RELINDX**2 ; 

CALL  SYMPUT ( ' REL ' , REL) ; 

END; 

PROC  RSQUARE  DATA=TEMP (TYPE=CORR)  NOPRINT  ADJRSQ  OUTEST  =  & RATING; 

%DO  M  =  1  %TO  &NVL2 ;  /*  SET  UP  A  MODEL  FOR  EACH  REMAINING  VAR  */ 
%LET  V  =  %SCAN (&VL2, &M)  ; 

%IF  &DELETION  %THEN  %LET  VLM  =  % REMOVE (&M, &VL2, &NVL2) ; 

%ELSE  %LET  VLM  =  &VL1  &V; 

&V  :  MODEL  &DEP  =  &LIST1  &VLM  /  INCLUDE=&NIV  STOP=&NIVl  ; 

%END ; 

DATA  &RATING;  SET  & RATING;  _ADJRSQ_  =  _AD JRSQ_/ &REL ; 

%END;  RUN; 

%MEND  STEP; 
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METASTEP.SAS 


% MACRO  METSTEP (RATINGS, LIST1, LIST2) ;  /*  THE  MAIN  META-ANALYSIS  */ 

/*  LIST1  =  LIST  OF  VARIABLES  THAT  ALWAYS  REMAIN  IN  REGRESSION  */ 

/*  LIST2  =  LIST  OF  VARIABLES  THAT  ARE  ADDED  OR  DELETED  */ 

%LET  NR  =  %NUMLIST (&RATINGS)  ; 

TITLE  'STEPWISE  META-ANALYSIS  FOR  BEST  MEAN  WHERRY-SHRUNKEN  R'; 

TITLE2  "&RATINGS"; 

TITLE3  'THE  FOLLOWING  PREDICTORS  REMAIN  IN  REGRESSION  AT  ALL  TIMES:'; 
TITLE4  "&LIST1" ; 

/*  VL1  =  THAT  PART  OF  LIST2  THAT  IS  IN  REGRESSION  DURING  ACCRETION 
OR  THAT  HAS  BEEN  DELETED  FROM  LIST2  DURING  DELETION  */ 

/*  VL2  =  THAT  PART  OF  LIST2  THAT  IS  NOT  YET  IN  REGRESSION  DURING  THE 
ACCRETION  PHASE,  OR  THAT  REMAINS  IN  REGRESSION  DURING  THE 
DELETION  PHASE  */ 

%LET  NLIST1  =  %NUMLIST (&LIST1) ;  %LET  NLIST2  =  %NUMLIST (&LIST2 ) ; 

%DO  DELETION  =  0  %TO  1;  /*  ACCRETION  PHASE  FIRST,  FOLLOWED  BY  DELETION*/ 
%LET  VL1  =  ;  %LET  VL2  =  &LIST2; 

%LET  NVL1  =  0;  %LET  NVL2  =  &NLIST2; 

%DO  PR  =  1  %TO  &NLIST2;  /*  LOOP  OVER  CANDIDATE  PREDICTORS  */ 

%IF  &DELETION  %THEN  %LET  NIV  =  %EVAL (&NLIST1  +  &NVL2  -  1) ; 

%ELSE  %LET  NIV  =  %EVAL (&NLIST1  +  &NVL1  +  1) ; 

/*  NIV  =  NUMBER  OF  PREDICTORS  CURRENTLY  IN  REGRESSION  */ 

%LET  NIV1  =  %EVAL (&NIV  +  1); 

%DO  IR  =  1  %TO  &NR; 

%LET  RATING  =  %SCAN (&RATINGS, &IR) ; 

%STEP (& RATING) 

%END; 

DATA  BASE  ;SET  & RATINGS; 

PROC  SORT  DATA=BASE ;  BY  _MODEL_; 

DATA  BASE;  SET  BASE;  IF  _ADJRSQ_  <  0  THEN  _ADJRSQ_  =  0; 

_ADJRSQ_  =  SQRT (_ADJRSQ_) ; 

PROC  MEANS  DATA=BASE  NOPRINT;  VAR  _ADJRSQ_;  WEIGHT  _EDF_;  BY  _MODEL_; 
OUTPUT  OUT=MBASE  MEAN=MEANRSQ ; 

DATA  MAXMEAN  ; SET  MBASE  END=LAST; 

RETAIN  MAXR  0  BESTV; 

FORMAT  NIV  4.  ; 

IF  MEANRSQ  >  MAXR  THEN  DO; 

MAXR  =  MEANRSQ; 

BESTV  =  _MODEL_;  END; 

IF  LAST  THEN  CALL  SYMPUT ( 'BESTV' , BESTV) ;  ELSE  DELETE; 

NIV  =  SYMGET('NIV') ; 

DROP  MEANRSQ  _MODEL_; 

PROC  APPEND  BASE=MAXR&  DE LET I ON  NEW=MAXME AN ; 

%LET  VL1  =  &VL1  &BESTV; 

%LET  VL2  =  %REMOVAL (&BESTV, &VL2)  ; 

%LET  NVL1  =  %EVAL (&NVL1  +  1)  ; 

%LET  NVL2  =  %EVAL (&NVL2  -  1)  ; 

%END;  ./*  PR  LOOP  OVER  PREDICTORS  */ 

PROC  PRINT  DATA=  MAXR&DELETION;  TITLE5  "MAXR&DELETION" ;  RUN; 

%END;  /*  ACCRETION /DELETION  LOOP  */ 

PROC  SORT  DATA=MAXR1;  BY  NIV; 

DATA  MAXR1 ; SET  MAXR1 ;  RENAME  MAXR=R_DEL  BESTV  =  V_DEL; 

DATA  MAXR; MERGE  MAXRO  MAXR1 ;  BY  NIV; 

RENAME  MAXR  =  R_ADD  BESTV  =  V_ADD ; 

PROC  PRINT  DATA=MAXR  NOOBS;  VAR  NIV  V_ADD  R_ADD  V_DEL  R_DEL; 

TITLE5  'COMBINED  ACCRETION  AND  DELETION  RESULTS';  RUN; 

DATA  MAXRO; SET  MAXMEAN;  IF  0;  /*  RESET  FOR  FUTURE  APPENDING  */ 

DATA  MAXR1 ; SET  MAXRO; 
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METASTEP.SAS 


%MEND  METSTEP; 

LET  ASVAB  =  GS1  AR1  WK1  PCI  N01  CS1  AS1  MK1  MCI  Ell  ; 

LET  AFQT  =  AR1  WK1  PCI  MK1; 

LET  NONAFQT  =  GS1  NOl  CS1  AS1  MCI  Ell; 

LET  ECAT  =  MCPCOR  SMPDCOR  IDPCOR  AOPCOR  T1MN  T2MN  SRPCOR  ORPCOR  TIDDT; 
LET  NOMOTOR  =  MCPCOR  SMPDCOR  IDPCOR  AOPCOR  SRPCOR  ORPCOR  ; 

LET  NOCOMP  =  AOPCOR  SRPCOR  ORPCOR  ; 

%LET  TLST  =  A11H5  B11H5  A13F1; 

%LET  FSGS  =  A11H5  B11H5  A13F1  APS1  ATC1  AC1  AE1  AMS1  AOl  AVI 

EMI  EN1  ET2  FC1  GM1  MM1  OS1  RM1; 

%LET  BE STD  =  A11H6  B11H9  A13F3  APS3  ATCX4  ATCY4 

AC 2  AE2  AMS2  A02  AV4  EM2  EN1  ET3  FC2  GM3  MM1  OS3  RM2  ; 
%LET  FSGS9=  A13F1  APS1  ATC1 

AC1  AE1  AMS1  AOl  AVI  ET2  OS1  ; 

%LET  BESTD9  =  A13F3  APS3  ATCX4  ATCY4 

AC2  AE2  AMS 2  A02  AV4  ET3  OS3 
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